Three names come up constantly when people ask what actually runs a model on local hardware: llama.cpp, Ollama, and vLLM. They are not three flavors of the same thing. One is the foundational engine, one is the friendly wrapper on top of it, and one is a completely different beast built for production traffic. Pick wrong and you either fight a serving framework you don’t need or hit a throughput wall you can’t escape. This guide gives you the crisp decision, the hardware and quantization realities, and a comparison table you can act on today.
The short version: single user on a home GPU, use Ollama. Serving an app to many concurrent users, use vLLM. Weird hardware, CPU-only, or you want to tune every knob, use llama.cpp directly. Everything below is why.
llama.cpp: the engine under everything
llama.cpp is the C/C++ inference engine that made local LLMs practical. It reads the GGUF format, a single portable file bundling weights, tokenizer, and metadata, and it runs on almost anything: NVIDIA (CUDA), AMD (ROCm), Apple Silicon (Metal), and other GPUs via Vulkan, plus pure CPU when you have no GPU at all. GGUF supports aggressive quantization down to roughly 2 bits per weight, which is why a 7-8B model fits in about 5-6 GB and a 70B squeezes into 40-45 GB at Q4.
It ships a real HTTP server too:
llama-server -m qwen3-8b-Q4_K_M.gguf -ngl 99 --port 8080The -ngl 99 offloads all layers to GPU. Drop it and the same model runs on CPU. That flexibility is the whole point. The cost is that you manage everything yourself: download the right GGUF, pick the quant, set context length and batch flags. It is the most control and the least hand-holding. If you enjoy tuning, or you’re on hardware nothing else supports, this is home.
Ollama: llama.cpp with the sharp edges filed off
Ollama is the friendly wrapper the majority of individuals should use, and it’s important to understand that it is built on llama.cpp’s GGML engine. It is not a competitor to llama.cpp so much as the ergonomic front door to it. Ollama adds the things llama.cpp deliberately leaves out: one-command model pulls, automatic hardware detection, quant selection by default, and a clean local API on port 11434.
ollama run qwen3:8bThat single line downloads a sensible quant, detects your GPU, loads the model, and drops you into a chat. The same server exposes an API you can hit from Python or any HTTP client, which is why so many local apps target it. And to answer the question everyone asks: yes, Ollama is free and open source.
The honest tradeoff: because Ollama picks defaults for you, raw llama.cpp with hand-tuned batch size and offload flags can be about 10-20% faster on a single model. For daily chat, coding, and small projects, you will never notice, and the DX win is enormous. Ollama is also not built to serve hundreds of simultaneous users, which is exactly where the third engine comes in. If you’re deciding between friendly desktop apps rather than engines, see Ollama vs LM Studio vs Jan.
vLLM: the production serving engine
vLLM is a different category. It exists to serve one model to many concurrent users at maximum throughput, and it is very good at it. Two ideas do the heavy lifting. PagedAttention stores the KV cache in non-contiguous memory blocks, like virtual memory paging, which kills the fragmentation that normally wastes GPU memory. Continuous batching dynamically slots new requests into the running batch as others finish, instead of waiting for a whole batch to complete. Add speculative decoding and tensor parallelism across multiple GPUs and you get a serving stack whose price-performance under load is hard to beat.
vllm serve Qwen/Qwen3-8B --dtype autoThat launches an OpenAI-compatible API server. The catches are real, though. vLLM is effectively GPU-only and hungry: it’s built around safetensors weights in FP16/BF16 or 4-bit quantization, so it wants enough VRAM to hold near-full or 4-bit models plus KV cache for every active conversation. Setup is heavier (Python environment, CUDA, more moving parts). For a single person asking one question at a time, all that batching machinery sits idle, and you’d have been happier with Ollama. vLLM earns its keep when the request count climbs.
The comparison table
| Engine | Best for | Hardware | Quant formats | Concurrency | Difficulty |
|---|---|---|---|---|---|
| llama.cpp | Tinkerers, weird/CPU hardware, max control | CPU, NVIDIA, AMD, Apple, Vulkan | GGUF (~2-8 bpw) | Low to moderate | Hard |
| Ollama | Individuals, single-user chat & coding | CPU + any supported GPU | GGUF (auto-selected) | Low to moderate | Easy |
| vLLM | Serving an app to many users | GPU-only, high VRAM | AWQ, GPTQ, FP8, INT4/INT8 | Very high | Moderate to hard |
Quantization: GGUF vs everything vLLM likes
This is where the split gets concrete. llama.cpp and Ollama live in GGUF land: Q4_K_M (~4.8 bpw) is the everyday sweet spot, Q8_0 for near-lossless, Q3_K_M when you’re desperate for VRAM. If you’ve been reading our GGUF quantization cheat sheet, that whole vocabulary belongs to the llama.cpp world.
vLLM speaks a different dialect. It’s built around AWQ and GPTQ (both compress weights to roughly 4-bit, cutting VRAM about 4x), plus FP8, which is the natural fit for H100/H200-class cards with native FP8 tensor cores and preserves quality with minimal calibration. GGUF support in vLLM exists but is experimental. Practically: if your workflow is GGUF files off Hugging Face, that’s a signal you belong on Ollama or llama.cpp, not vLLM.
The decision, made simple
- You’re one person on a home GPU (an RTX 3060, 4090, whatever) and want to chat, code, or prototype. Use Ollama. It’s the default for a reason. Match your model to your card with our coding model by VRAM guide.
- You love tuning, run exotic or CPU-only hardware, or need to control every flag. Use llama.cpp directly. Ollama can’t expose everything, and you’ll want the raw server.
- You’re putting a model behind an app and expect real concurrent traffic. Use vLLM. Nothing else here handles dozens of simultaneous requests without melting VRAM.
The contrarian point nobody says plainly: this is rarely an either/or, and vLLM is overkill for almost every hobbyist. Because Ollama is llama.cpp underneath, “Ollama vs llama.cpp” is really “convenience vs control” over the same engine. And spinning up vLLM to talk to yourself is like renting a semi-truck for the groceries. Start with Ollama; graduate to vLLM only when a real user count forces you to.
Which one powers your setup
If your goal is private, local AI you actually use every day, Ollama is the engine to start with, the friendly door onto llama.cpp, free, and the API most local apps target.
And if the thing you were planning to build on that engine is an AI companion, be honest about whether you want the project or the companion. Ember is the instant version: an uncensored 18+ companion that ships its own engine and weights, so there is no GGUF to fetch and no runtime to choose, running on your own NVIDIA card. Pick the engine that fits your hardware, for almost everyone, that’s Ollama.