If you’ve been shopping for a GPU or a mini PC to run a local AI companion, you’ve seen the number everywhere: tokens per second. Benchmarks brag about it, Reddit argues about it, and somewhere a spec sheet implies you need 100 tok/s or your setup is garbage. Here’s the honest answer almost nobody gives you plainly: for a back-and-forth chat you actually read, the speed that matters is roughly how fast you read. Once your model outputs text faster than your eyes consume it, extra speed stops mattering for the experience. This guide explains what tok/s really means, where the usable floor is, why the first token can matter more than the rest, and exactly what to expect from each hardware tier.
What “tok/s” means in plain terms
A token is a chunk of text the model generates one piece at a time. It’s not quite a word and not quite a letter, it’s a sub-word unit. As a rough rule of thumb, 1 token ≈ 0.75 English words, or about 4 characters. So “100 tokens” is roughly 75 words, or a solid paragraph.
Tokens per second (tok/s) is simply how many of those chunks the model produces each second during generation. If a model runs at 10 tok/s, it’s writing about 7-8 words every second. That’s the generation (decode) speed, the part you watch stream onto the screen.
The key mental shift: tok/s is a throughput number, not a quality number. A blazing-fast model isn’t smarter; it just types faster. What you care about as a chatter is whether the typing keeps up with you, and that bar is lower than the benchmark culture suggests.
The usable floor: human reading speed (~7-10 tok/s)
Here’s the anchor for the whole question. The average adult reads English at roughly 200-300 words per minute. Convert that to tokens:
- 250 words/min ÷ 60 = ~4.2 words/sec
- ÷ 0.75 words per token = ~5.5 tokens/sec to match a comfortable reading pace
So purely on reading mechanics, ~5-6 tok/s is the absolute floor where text arrives as fast as you can read it. But “as fast as I can read” isn’t the same as “feels good.” A little headroom makes the difference between waiting and flowing. In practice:
| Speed | What it feels like |
|---|---|
| Under 3 tok/s | Painful. You finish reading and stare at a blinking cursor. Fine for batch jobs, not for chat. |
| 3-5 tok/s | Tolerable for offline/private work where you’d accept the trade. Slightly behind a fast reader. |
| 7-10 tok/s | The usable sweet spot for chat. Text streams a touch faster than you read, natural, conversational. |
| 15-30 tok/s | Comfortably fast. You never wait; you skim if anything. |
| 40+ tok/s | Faster than you can read. Great for long generations and code, but invisible “extra” for plain chat. |
Bottom line: 7-10 tok/s is the practical minimum for a chat that feels good, and anything above ~15 tok/s is gravy for conversational use. Chasing 60 tok/s for a roleplay session is optimizing a number you’ll never feel. This is the same conclusion we reach in our deeper dive on whether tokens per second is even worth obsessing over, the honest answer is that “usable” plateaus early.
Streaming vs total latency: why first-token matters
Average tok/s hides the single biggest thing people actually feel: time to first token (TTFT), the gap between hitting Enter and the first word appearing.
Two metrics, two very different experiences:
- Time to first token (TTFT): how long until the response starts. This is the “is it frozen?” feeling.
- Inter-token latency / decode speed: the steady tok/s once it’s rolling, the streaming speed your eyes track.
A model can have a great average tok/s but a sluggish TTFT, and it’ll feel slow because you sat watching nothing for two seconds. Conversely, a modest 9 tok/s with near-instant first token feels snappy and alive, because streaming lets you start reading the moment generation begins, you read the opening words while the rest is still being written.
What drives TTFT:
- Prompt length (context). The model must process your entire prompt, system prompt, character card, chat history, before it writes word one. This is the prefill stage. Long roleplay histories and big system prompts inflate TTFT noticeably. A 4,000-token history means thousands of tokens to chew through first.
- Cold start. The first message after loading a model is slower while weights page into VRAM/RAM. Subsequent replies are quicker.
- Memory bandwidth. More on this below, it’s the real bottleneck for local LLMs.
Practical takeaways: keep your context lean (trim bloated system prompts, cap history length), and value a setup that streams tokens as they generate rather than waiting for the full reply. Streaming is why 9 tok/s on a local box can feel better than a “fast” cloud app that buffers the whole message before showing it.
Tok/s by hardware tier (quick reference)
Local LLM speed is governed less by raw compute and more by memory bandwidth, how fast the chip can move model weights. That’s why GPUs (huge bandwidth) crush CPUs, and why Apple Silicon punches above its weight (unified memory with high bandwidth). The other lever is model size: a 7B-8B model at a Q4_K_M quantization is several times faster than a 70B model on the same hardware.
Here’s a realistic, category-level picture for a mid-size chat model (think 7B-13B class, 4-bit quantized). Treat these as ballpark ranges, not promises, exact numbers depend on the specific model, quant, and context length:
| Hardware tier | Example | Realistic tok/s (7B-13B, Q4) | Verdict for chat |
|---|---|---|---|
| Modern dedicated GPU, 12-24 GB | RTX 4070 / 4080 / 4090, etc. | Tens to 100+ tok/s | Effortless. Way past the floor. |
| Entry/older GPU, 8 GB | Budget 8 GB cards | Comfortably 20-50+ tok/s on an 8B model | Great for chat; size your model to VRAM. |
| Apple Silicon (unified memory) | M-series Mac / Mac mini | Often 10-40+ tok/s depending on chip + model | Smooth, quiet, efficient. |
| CPU only (no GPU) | Modern desktop CPU + RAM | Low single digits to ~10 tok/s on small models | Usable for small models; tight on larger ones. |
If you want the full breakdown of what to buy and why, our local AI hardware guide maps models to VRAM and budgets in detail, and the question of whether you need a GPU at all for an AI companion is more nuanced than “buy a 4090.”
The headline: almost any modern dedicated GPU clears the usable bar for chat with room to spare. The interesting case is the no-GPU path.
When “slow but offline” is an acceptable trade
There’s a real category of user for whom 3-6 tok/s on a CPU is a perfectly good deal, and it’s worth naming honestly rather than pretending everyone needs a 4090.
You’re in this camp if:
- Privacy is the point. A slightly slower reply that never leaves your machine beats an instant reply logged on someone’s server. If your reason for going local is that the conversation is nobody’s business but yours, a few seconds of latency is a rounding error against that.
- You’re reading, not racing. Thoughtful, slower-paced chat (journaling, companionship, creative writing) doesn’t need machine-gun output. You’re savoring the reply, not speed-running it.
- You have RAM but no GPU. A modern laptop or mini PC with 16-32 GB of RAM can run a small-to-mid model on CPU. It won’t win benchmarks, but it’ll hold a conversation. Our guide to running local AI without a GPU covers how to pick a model small enough to stay responsive.
The trade only goes bad when slowness plus a big context history pushes TTFT into “did it crash?” territory. The fix is to run a smaller, well-quantized model (an 8B at Q4_K_M instead of a 13B), keep context tight, and accept that you’re trading a little brilliance for a lot of privacy and zero subscription. For many people, that’s exactly the right trade. If you’re unsure your machine can even hold up, can my PC run an AI companion walks through the honest minimum.
Your rig fast enough? Run it locally
If you’ve got a dedicated GPU, even a humble 8 GB card, you are almost certainly past the usable floor for a great chat experience. The path is well-trodden:
curl -fsSL https://ollama.com/install.sh | sh
ollama run <model>Ollama serves a local API on 127.0.0.1:11434 (loopback only, nothing leaves your box), and you point a companion front-end at it. That’s the whole pitch of running local: the model lives on your hardware, the conversation stays on your hardware, and there’s no monthly bill metering your tokens. If you’re new to the workflow, start with how to run AI locally.
If your rig clears the bar, and most modern GPUs do, the guides above turn that throughput into an actual companion instead of a terminal benchmark. Ember is the packaged version of exactly that: an uncensored 18+ companion whose brain is sized for an 8 GB NVIDIA card, so the tok/s you just measured is the speed she answers at. $29 paid once in crypto, running entirely on your own box.
Too slow? Speed it up locally
Be honest with yourself about the measurement. If you ran the numbers and your CPU-only box is grinding out 2-3 tok/s with a long history, forcing an oversized model will make the experience worse, not more private-feeling. A companion that takes 30 seconds to start replying isn’t a companion; it’s a loading screen.
The fix is to right-size locally rather than give up on privacy: drop to a smaller, faster model, a well-tuned 7-8B often feels snappier than a sluggish 13B, or add even a modest GPU, which is the single biggest jump in tok/s you can buy. Both keep the entire companion on your own machine, which is the whole point.
How to measure your own tok/s
Don’t guess, measure. It takes two minutes and ends the speculation.
With Ollama, run any model and add the --verbose flag, then send one message:
ollama run <model> --verboseAfter each response, Ollama prints timing stats including eval rate (that’s your decode tokens/second) and prompt eval timing (which feeds your time-to-first-token). The eval rate line is the number to compare against the 7-10 tok/s floor.
A quick manual sanity check if you’re using a GUI: count roughly how many words appear in ~5 seconds of streaming. Multiply by 12 (5s → per-minute), then divide by ~45 to get tok/s in your head, or just eyeball it: if the text streams a little faster than you naturally read, you’re in the good zone. If you finish each line and wait, you’re under the floor and should switch to a smaller or more aggressively quantized model.
Things that move your number, in order of impact:
- Model size, drop from 13B to 8B for a big jump.
- Quantization,
Q4_K_Mis the popular speed/quality balance; lighter quants run faster. - Context length, shorter prompts and trimmed history cut TTFT.
- Offload, keep the whole model in VRAM if you can; spilling to system RAM tanks speed.
Measure once, pick a model that comfortably clears 7-10 tok/s on your hardware, and stop worrying about the leaderboard numbers. Usable is usable.
Once you know your real tok/s, the decision gets easy: if your machine clears the bar, run the whole companion locally and privately, the entire stack lives on your box. If it doesn’t clear the bar yet, right-size the model or add a modest GPU: an 8 GB NVIDIA card is roughly where a companion stops feeling like a benchmark. Above that line, Ember is the assembled version, uncensored, 18+, fully local, $29 paid once in crypto with no card on file.