If you’ve ever stared at a Hugging Face model page with twenty different .gguf files, Q2_K, Q4_K_S, Q4_K_M, Q5_K_M, Q6_K, Q8_0, IQ4_XS, and had no idea which one to download, this page is the answer. Quantization is just compression for model weights: it shrinks a model so it fits in your VRAM, trading a little quality for a lot of size. The whole game is picking the largest quant that still fits entirely on your GPU. Get that right and you’ll get near-original quality at full speed. Get it wrong and you either waste quality you didn’t need to lose, or you spill into system RAM and watch your tokens-per-second collapse.
Here’s the 30-second version, then the details if you want them.
The 30-Second Rule: Largest Quant That Fully Offloads Wins
There is one rule that resolves 90% of quant decisions:
Pick the biggest quant whose file size leaves ~1-2 GB of VRAM headroom for context. If it all fits on the GPU, you’re done.
Why this works: an LLM running on a GPU is fast because every weight lives in VRAM. The moment a model is too big and some layers get offloaded to CPU/system RAM, generation speed falls off a cliff, often from 40+ tokens/sec down to single digits. A smaller quant that runs fully on the GPU will beat a larger quant that spills, both on speed and usually on real-world usefulness.
So the decision isn’t “what’s the highest-quality quant?” It’s “what’s the highest-quality quant that fully offloads?” For most people on a single consumer GPU, the answer lands on Q4_K_M or Q5_K_M, and you can stop reading here if you just need a default. The rest of this page is for sizing it exactly.
If you’re still choosing hardware, our local AI hardware guide covers what VRAM actually buys you.
K_M vs K_S vs I-Quants Explained Simply
The cryptic suffixes encode two things: how many bits per weight and which strategy was used.
- The number (
Q4,Q5,Q6,Q8) = the average bits per weight. Higher = bigger and more faithful to the original model. _K= a K-quant, the modern default. K-quants group weights into super-blocks of 256 and store separate scaling factors per sub-block, spending more bits on the layers that matter most. This is why a 4-bit K-quant beats an old “legacy” 4-bit quant of the same size._S/_M/_L= Small, Medium, Large variants of the same bit level._Mkeeps a few critical tensors (like attention layers) at higher precision;_Ssqueezes them down to save space._Mis almost always the right pick,_Ssaves a few hundred MB for a measurable quality hit.IQ(I-quants) = a newer family (IQ2_XS,IQ3_M,IQ4_XS,IQ4_NL…) that uses an importance matrix (imatrix), a calibration pass that figures out which weights matter, to preserve quality at very low bitrates. AnIQ4_XSis smaller thanQ4_K_Myet, when calibrated well, can match it.
The practical rule of thumb:
| If you… | Use |
|---|---|
| Fit a normal K-quant comfortably | Q-K quants (Q4_K_M, Q5_K_M), simpler, robust |
| Are squeezing a model that barely fits | I-quants (IQ4_XS, IQ3_M), more quality per GB |
| Run on CPU or a weak GPU | Lean toward K-quants, I-quants can be slower to decode on some hardware |
I-quants buy you the most quality per gigabyte, which matters most at the low end (2-3 bit) or when you’re trying to cram a 70B model into limited VRAM. For everyday 4-5 bit use on a GPU that fits the model, plain K-quants are the safe default.
Quality vs Size by Quant Level
This is the table to bookmark. Sizes are relative multipliers, multiply by the model’s parameter count in billions to get a rough GB figure (the estimator below makes this exact).
| Quant | ~Bits/weight | Size vs fp16 | Quality | Verdict |
|---|---|---|---|---|
| Q2_K | ~2.6 | ~16% | Noticeably degraded | Last resort to make a big model fit at all |
| Q3_K_M | ~3.4 | ~21% | Usable, some slips | Squeeze tier for large models |
| IQ3_M | ~3.5 | ~22% | Better than Q3_K at same size | Best low-bit option if imatrix is good |
| Q4_K_M | ~4.5 | ~28% | Very good, the sweet spot | Default for most people |
| IQ4_XS | ~4.3 | ~26% | ≈ Q4_K_M, slightly smaller | Great when you need a hair more room |
| Q5_K_M | ~5.5 | ~34% | Near-imperceptible loss | Upgrade if it still fits |
| Q6_K | ~6.6 | ~41% | “Almost lossless” | Diminishing returns begin here |
| Q8_0 | ~8.5 | ~53% | Essentially lossless (~0.01 perplexity vs fp16) | Only when VRAM is plentiful |
| fp16/bf16 | 16 | 100% | The original | Rarely worth it for inference |
The honest takeaway: the quality difference between Q4_K_M and Q8_0 is small for chat, roleplay, and most writing. The difference between Q4_K_M and Q2_K is large. Spend your VRAM getting to Q4_K_M first; spend what’s left climbing toward Q5/Q6.
Copy-Paste Size Estimator
You don’t need to memorize file sizes. Multiply billions of parameters × bits-per-weight ÷ 8 to get GB, then add ~10-20% overhead, plus room for context (KV cache).
Drop this into a terminal:
# usage: ./quantsize.sh <params_in_billions>
# prints approximate file size for common quants
params=${1:-8}
declare -A bpw=( [Q3_K_M]=3.4 [Q4_K_M]=4.5 [IQ4_XS]=4.3 [Q5_K_M]=5.5 [Q6_K]=6.6 [Q8_0]=8.5 )
echo "Model: ${params}B parameters"
printf "%-9s %8s %12s\n" "QUANT" "FILE_GB" "FITS_8GB?"
for q in Q3_K_M Q4_K_M IQ4_XS Q5_K_M Q6_K Q8_0; do
gb=$(awk -v p="$params" -v b="${bpw[$q]}" 'BEGIN{printf "%.1f", p*b/8*1.1}')
fit=$(awk -v g="$gb" 'BEGIN{print (g<6.5)?"yes":"tight/no"}')
printf "%-9s %8s %12s\n" "$q" "$gb" "$fit"
doneRun bash quantsize.sh 8 for an 8B model. The ×1.1 is overhead; you still need 1-2 GB on top for context. A quick mental shortcut: a 4-bit (Q4_K_M) model is roughly half its parameter count in GB, an 8B is ~4.5 GB, a 13B is ~7.5 GB, a 70B is ~40 GB.
Which Quant for Which VRAM
Match the model+quant to your card so it fully offloads. These assume ~1-2 GB reserved for context and your desktop.
| VRAM | Comfortable pick | What it runs well |
|---|---|---|
| 6 GB | 7-8B @ Q4_K_M or IQ4_XS | Solid 8B chat; tight on context |
| 8 GB | 8B @ Q4_K_M / Q5_K_M | The mainstream sweet spot, see best local LLM for 8GB VRAM |
| 12 GB | 8B @ Q6_K, or 13-14B @ Q4_K_M | Bigger models or higher quant, your call |
| 16 GB | 14B @ Q5_K_M, or 22B @ Q4_K_M | Comfortable headroom |
| 24 GB | 32-34B @ Q4_K_M, or 14B @ Q8_0 | Genuinely capable local models |
| 48 GB+ | 70B @ IQ4_XS/Q4_K_M | Frontier-class local, see VRAM for a 70B model |
Once you’ve picked a model, pulling it is one line with Ollama, which auto-selects a sensible quant:
ollama run llama3.1:8b # default tag, usually Q4_K_M
ollama run qwen2.5:14b-instruct-q5_K_M # pin an explicit quantIf Ollama isn’t set up yet, start with how to run AI locally.
When Q8 or fp16 Is Worth It
For chat and roleplay, almost never, the gap over Q5/Q6 is too small to feel, and you’re paying double the VRAM. Reach for Q8_0 only when:
- You have VRAM to burn and the model still fits with full context. Free quality is free quality.
- The task is precision-sensitive: code generation, math, structured/JSON output, function-calling, or long multi-step reasoning where small errors compound.
- You’re building a dataset or doing fine-tuning prep, where you want the cleanest possible base behavior.
fp16/bf16 is almost never worth it for inference, Q8_0 is within a rounding error of it at half the size. fp16 matters when you’re training or fine-tuning, not when you’re chatting. If you find yourself reaching for Q8 to “fix” a model that feels dumb, the problem is usually the model choice or your sampling settings, not the quant.
Common Quant Mistakes
- Picking a quant that spills off the GPU. The #1 error. A
Q6_Kthat offloads two layers to CPU is slower and feels worse than aQ4_K_Mthat runs fully on the card. Fit first. - Forgetting the KV cache. Long context eats VRAM separately from weights. A model that fits at 2K context can OOM at 32K. Always leave headroom.
- Choosing
_Sto save a few hundred MB. The quality hit isn’t worth it. Use_M. - Going below Q4 when you didn’t have to. Q2/Q3 are squeeze tiers for when a model otherwise won’t fit at all, not a default. Prefer a smaller model at Q4 over a bigger model at Q2.
- Using a non-imatrix I-quant. I-quants depend on a good importance matrix. An
IQ4_XSfrom a careless uploader can underperform; from a trusted one it shines (next section). - Assuming bigger quant = smarter. Above Q5/Q6 you’re paying real VRAM for changes you can’t perceive in normal use.
Where to Find Trustworthy Quants
Quantization quality varies by who made the file, especially for I-quants that need a calibration pass. Two uploaders are the community standard on Hugging Face:
- bartowski, fast, well-labeled repos with the full ladder of K-quants and imatrix I-quants for nearly every major model. The README usually includes a quality/size chart so you can pick at a glance.
- mradermacher, enormous coverage, including static quants (
-GGUF) and imatrix quants (-i1-GGUF). If a model exists, mradermacher has probably quantized it.
For I-quants specifically, look for the i1- prefix or “imatrix” in the repo name, that’s the calibrated version. Before downloading anything, sanity-check the source: a quant is just a binary blob, so provenance matters. Our guide on downloading GGUF models safely from Hugging Face covers what to verify.
Once you’ve internalized the 30-second rule, picking a quant really is a 30-second decision: estimate the size, match it to your VRAM, grab the _M K-quant (or an imatrix I-quant when you’re tight), and run. The harder question is usually the one before the quant, which model, on which hardware, for what. And if the thing you’re actually shopping for is an AI companion rather than a model collection, Ember skips the whole exercise: her model and quant are already chosen and tuned, sized to fit an 8 GB NVIDIA card, so there is no cheat sheet to apply.