You’re on a Hugging Face model page, there are fifteen .gguf files staring back at you, and the only honest question is: which one will I actually notice a difference with? Here’s the short answer most guides bury: for the overwhelming majority of people, download Q4_K_M and stop. It’s the standard “best quality-per-byte” pick for a reason, the quality loss versus the full-precision model is small enough that you won’t feel it in chat, roleplay, or everyday writing, and it fits hardware that bigger quants don’t. Everything past that is fine-tuning. This guide is about what you’ll actually perceive at each quant level, where the diminishing returns kick in, why a 7B model punishes aggressive quantization far harder than a 70B does, and how to pick by your VRAM and model size without overthinking it.
If you want the raw format definitions, what _K_M means, the size math, the suffix decoder, that lives in the companion GGUF quantization cheat sheet. This page is the quality decision: not “what is a Q5,” but “should I bother.”
Quantization in one sentence (and what it costs)
Quantization compresses a model’s weights from 16 bits each down to roughly 8, 5, 4, 3, or even 2 bits. Fewer bits means a smaller file that fits in less VRAM, and a small, usually-graceful loss of fidelity. The number in the quant name (Q4, Q5, Q8) is approximately the average bits per weight. Higher number, bigger file, closer to the original model.
The thing nobody tells you plainly: does quantization hurt quality? Yes, but the curve is steeply non-linear. Going from full precision down to 8-bit costs you almost nothing. Eight to five costs you a little. Five to four costs you a little more but is still hard to feel. Below four, the cost starts climbing fast, and below three it can fall off a cliff. So the entire game is finding the point where you’ve stopped losing anything you’d notice, and that point, for most models on most hardware, is right around Q4_K_M.
What you actually notice at each level
Forget exact perplexity figures for a moment (we’ll get to why those are slippery). Here’s the practical, felt difference at each tier, treat these as directional impressions, not guarantees, because the exact behavior depends on the model and the task.
| Quant | ~Bits | What you’ll actually notice | Honest verdict |
|---|---|---|---|
| Q8_0 | ~8 | Indistinguishable from the original in normal use | Lossless-for-practical-purposes; rarely worth the VRAM |
| Q6_K | ~6.5 | No perceptible difference for chat/writing | Where diminishing returns are already obvious |
| Q5_K_M | ~5.5 | A near-imperceptible step down from Q6 | Great “I have the room” upgrade over Q4 |
| Q4_K_M | ~4.8 | Very occasional, minor slips; mostly invisible | The default. Start here. |
| Q3_K_M | ~3.9 | More word-choice errors, weaker long reasoning | Squeeze tier, only to make a model fit |
| Q2_K | ~2.6 | Noticeable: repetition, logic slips, format breaks | Last resort to run a big model at all |
The pattern to internalize: the gap between Q4_K_M and Q8 is small, for chat and roleplay it’s often nothing you’d pick out in a blind test. The gap between Q4_K_M and Q2_K is large and shows up as the model getting visibly dumber: it loses the thread in long conversations, fumbles instructions it should nail, and occasionally produces malformed output (broken JSON, ignored formatting, repeated phrases).
This is why the advice “spend your VRAM getting to Q4 first, then climb” holds. A jump from Q2 to Q4 buys real, felt quality. A jump from Q5 to Q8 buys you mostly peace of mind.
Why Q2 and Q3 hurt more than the numbers suggest
There’s a trap in reading quantization charts: the perplexity increase from Q5 to Q4 and from Q4 to Q3 can look like similar small steps on paper, but they don’t feel the same. Low-bit quantization doesn’t degrade quality evenly across everything the model does. It tends to leave casual chat mostly intact while quietly damaging the harder stuff first, multi-step reasoning, precise instruction-following, code, math, and structured output. So a Q3 model can seem fine in a quick “tell me a joke” test and then fall apart on the task you actually downloaded it for.
That’s also why benchmark perplexity numbers should be read as a small-but-measurable rise as you drop bits, not as a precise quality dial. Perplexity is an average over a text corpus; it under-weights exactly the rare, high-stakes tokens (the right variable name, the correct logical step) that low-bit quants are most likely to get wrong. Trust the direction of those numbers, lower bits, higher perplexity, worse quality, but don’t treat a tiny perplexity gap as proof two quants feel identical on your workload.
The part most guides skip: small models suffer more
Here is the single most useful thing to understand about quantization quality, and it changes how you should shop: larger models tolerate aggressive quantization far better than small ones.
A big model carries a lot of redundancy. Squeezing its weights to 4 bits knocks out precision the model can largely route around, the knowledge is spread across so many parameters that rounding each one a little does limited damage. A small model has no such slack. Every weight is doing more work, so the same 4-bit rounding removes a larger share of what the model actually knows. The research direction here is consistent: the perplexity rise from quantization is modest for larger models and noticeably steeper for very small ones.
In plain terms:
- A 70B at Q4_K_M stays remarkably close to its full-precision self. This is why running a 70B model in limited VRAM at 4-bit is a genuinely good deal, not a sad compromise.
- A 7-8B at Q4_K_M is still very usable and the right default for that size, but it’s giving up proportionally more than the 70B did.
- A 1-3B model at Q4 is where you start to see real fraying. For tiny models, lean toward Q5, Q6, or even Q8 if you can, the extra bits matter most exactly where the model has the least to spare. The good news is small models are small, so a higher quant costs you very little VRAM.
The corollary is the rule that resolves most “which file?” debates: a bigger model at a lower quant almost always beats a smaller model at a higher quant, as long as both fully fit. A 70B at Q4 will out-think an 8B at Q8 on essentially everything, despite not using dramatically more memory. So if you’re torn between “32B at Q4” and “14B at Q6,” the 32B usually wins, provided it fits without spilling.
The VRAM-versus-quality tradeoff (and the cliff under it)
Every extra bit of quantization is VRAM you have to find. The reason Q4_K_M dominates isn’t that it’s the highest quality, it’s that it’s the best quality that still fits on common hardware. And fitting matters more than quant level because of a hard performance cliff.
When a model is even slightly too big for your VRAM, some of its layers get offloaded to system RAM, and generation speed collapses, often from comfortable double-digit tokens per second down to a crawl. A Q5 that spills two layers onto the CPU is both slower and effectively worse to use than a Q4 that runs entirely on the GPU. So the real priority order is:
- Fit the model fully in VRAM (weights plus room for context, the KV cache grows with conversation length and eats VRAM separately).
- Then climb to the highest quant that still fits with that headroom.
If you’re not sure whether your weights-plus-context will fit, the mechanics of VRAM and the KV-cache tax are worth understanding before you download. System RAM is not a substitute here, spilling onto it is exactly what triggers the speed collapse, so “just add more RAM” doesn’t rescue a model that’s too big for the GPU.
Where imatrix (I-quants) fit
You’ll also see files with names like IQ4_XS, IQ3_M, or repos tagged i1-. These are importance-matrix quants. The uploader runs a calibration pass that identifies which weights matter most, then spends the limited bit budget protecting those. The payoff: more quality per gigabyte, and the effect is largest at low bitrates, exactly where standard quants hurt most.
How to think about them for a quality decision:
- At 4-bit and up where the model fits comfortably: plain K-quants (
Q4_K_M,Q5_K_M) are the simple, robust default. I-quants help least here because there’s already little to lose. - When you’re squeezing, 2-bit and 3-bit territory, or cramming a big model into too little VRAM: I-quants earn their keep. An
IQ3_Mcan hold together noticeably better than aQ3_Kof similar size. If you must go low-bit to fit, prefer the imatrix version. - Quality depends on who built the file. An I-quant from a careless uploader, with a weak or missing calibration pass, can underperform a plain K-quant. From a trusted source it shines. The cheat sheet lists the community-standard uploaders to look for.
- One tradeoff: I-quants can be slower to decode on some CPUs and weaker GPUs. On a card that fits the model anyway, plain K-quants are often the smoother choice.
So imatrix isn’t a free upgrade you slap on everything, it’s the tool you reach for specifically when you’re bit-starved.
Rule of thumb: pick by VRAM and model size
Match the largest model+quant that fully offloads, reserving roughly 1-2 GB for context. These are starting points, not laws, exact fit depends on context length and the specific model.
| Your VRAM | Sensible default | Quality note |
|---|---|---|
| 8 GB | 7-8B @ Q4_K_M | The mainstream sweet spot, see best local LLM for 8GB VRAM. Bump to Q5 only if context allows |
| 12-16 GB | 8B @ Q6/Q8, or 13-14B @ Q4_K_M | Room to either raise the quant or the model |
| 24 GB | 32B @ Q4_K_M, or 14B @ Q8 | Prefer the bigger model at Q4 over the smaller at Q8, see best local LLM for 24GB |
| Tiny models (1-3B) | Q6_K or Q8_0 | The exception: small models need the extra bits, and they’re cheap |
| Barely-fits big model | IQ-quant at 3-4 bit | Where imatrix and low-bit are worth it |
A few decisions this resolves directly:
- Roleplay and chat are forgiving. These tolerate Q4 (sometimes even Q3) better than code or math, because a slightly less precise word choice rarely breaks immersion. If your use case is conversation, see best local LLMs for roleplay, don’t burn VRAM chasing Q8 you won’t feel.
- Code, math, JSON, and tool-calling are not forgiving. Small errors compound and break output. If that’s your workload and the model fits, Q5_K_M or Q6_K is a reasonable upgrade over Q4.
- MoE models change the VRAM math. A mixture-of-experts model’s total size still has to fit even though only some experts run per token, so the same quant rules apply to the full footprint, how MoE models fit in low VRAM covers the wrinkle.
When Q8 (or full precision) is actually worth it
Rarely, and for specific reasons, not as a general “more is better” instinct:
- You have VRAM to spare and the model fits at full context anyway. Free quality is free quality; take it.
- The task is precision-sensitive: code generation, math, strict structured output, or long multi-step reasoning where one wrong token derails everything.
- You’re building datasets or prepping a fine-tune and want the cleanest base behavior.
Full precision (fp16/bf16) is almost never worth it for inference, Q8 sits within a rounding error of it at half the size. fp16 matters when you’re training, not chatting. And if a model feels dumb and you’re reaching for a higher quant to “fix” it, the culprit is usually the model choice or your sampling settings, not the quantization.
The bottom line
Download Q4_K_M and you’ll be right the vast majority of the time. Climb to Q5 or Q6 when the model fits with room to spare and your task rewards precision. Drop to Q3 or an imatrix low-bit quant only when it’s the difference between running a bigger, smarter model and not running it at all, and bias small models upward, because they have the least to give. Above all, fit the whole model in VRAM before you chase quant levels: a quant that spills off the GPU loses more, in speed and felt quality, than any bit you saved. Get the model choice right first, pick the largest quant that fully fits, and the difference between the “perfect” quant and the practical one will be a difference you can’t feel.