If you’ve ever had a local AI companion go from charming to robotic mid-conversation — repeating the same phrase, forgetting your name, suddenly sounding like a customer-service bot — the model usually gets the blame. Often it’s not the model. It’s the backend: the program that actually loads the weights, samples each token, and decides what to do when your chat outgrows the context window. The same GGUF file can feel sharp and characterful on one backend and flat on another. KoboldCpp and Ollama are the two backends most people reach for, and they make very different trade-offs. This guide walks through how each one shapes the feel of roleplay and companion chat — not just tokens per second — so you can pick the one that fits how you actually talk to your AI.
Why the backend matters for companion feel, not just speed
A backend does three jobs that directly shape personality:
- Sampling — choosing the next token from the model’s probability distribution. The sampler settings (temperature, repetition penalty, and newer tricks like DRY and XTC) are the difference between vivid, varied prose and a character that loops the same sentence structure forever.
- Context handling — what happens when the conversation gets longer than the model’s context window. Naive truncation drops the oldest messages and can wipe out the part of the chat where your companion learned who you are.
- Prompt assembly — how your system prompt, character card, and history get stitched into the final prompt the model sees.
Speed matters, but both backends sit on top of the same llama.cpp inference core, so raw throughput is broadly comparable on the same hardware and quant. The real divergence is in control over sampling and context — and that’s exactly what determines whether a companion feels consistent and alive over a long session. (If you’re still choosing a model to run on either backend, start with the best local LLMs for roleplay.)
KoboldCpp: samplers, context shifting, GGUF flexibility
KoboldCpp is a single-file, zero-install binary built specifically for the GGUF + roleplay crowd. It ships a built-in web UI (KoboldAI Lite) with chat, adventure, instruct, and story-writer modes, loads Tavern character cards directly, and — crucially — exposes the full sampler stack.
What you get over a plainer backend:
- Full sampler control, including modern anti-repetition samplers. KoboldCpp supports DRY (Don’t Repeat Yourself) and XTC (Exclude Top Choices) alongside the classics. DRY is a smarter repetition penalty that punishes repeated sequences rather than individual tokens, and XTC nudges the model off its most predictable choices to keep prose fresh — both are big levers for roleplay samplers that fight the “she smiled. she smiled. she smiled.” death spiral.
- Context Shifting. When your chat exceeds the context window, KoboldCpp can shift the KV cache instead of reprocessing the whole prompt — trimming from the top while preserving recent turns, with minimal quality impact and no slow re-ingest on every message. For long companion sessions this is the single most important feature, and it’s the thing people mean when they talk about context shifting roleplay. (Note: it depends on a modern GGUF and is bypassed if you inject content mid-context; older formats fall back to the weaker SmartContext.)
- GGUF flexibility. You point it at any GGUF file on disk — any quant, any source from Hugging Face, any fine-tune — and tune
--contextsize,--gpulayers, FlashAttention, and more by hand. No model registry, no re-packaging.
The cost of all that control is that KoboldCpp hands you the knobs and expects you to turn them. That’s a feature for tinkerers and a tax for everyone else.
Ollama: simplicity, app-backend, scripting
Ollama optimizes for a different goal: make a local model trivially easy to run and to build on top of. One install, then:
curl -fsSL https://ollama.com/install.sh | sh
ollama run llama3.1It pulls models from a curated registry, manages them like Docker images, and exposes a clean HTTP API on the loopback address 127.0.0.1:11434. That API is the whole point: Ollama is less a roleplay UI than an app backend — the thing a desktop companion app, a script, or a custom frontend talks to. (See how to install Ollama for the full walkthrough, and Ollama vs LM Studio vs Jan for how it compares to other easy launchers.)
Where Ollama shines:
- Setup is genuinely two commands. Defaults are sane; you don’t have to know what a sampler is to get a coherent reply.
- Scripting and integration. The stable REST API and official Python/JS libraries make it the easy choice when you’re programming against a model rather than chatting in a UI.
- Model management.
ollama list,ollama pull, andModelfiles let you pin system prompts and parameters into a named model you can reuse.
The trade-offs for roleplay specifically: the registry favors instruct-tuned and SFW models, so for uncensored companion work you’ll often import a GGUF via a custom Modelfile (covered in Ollama uncensored models). And while you can set temperature, top_p, and repeat_penalty, Ollama doesn’t expose the deeper sampler stack — there’s no native DRY/XTC, and no KoboldCpp-style context shifting. When the window fills, you’re relying on the client’s truncation strategy.
Roleplay quality comparison: consistency, repetition, memory
Same model, same quant — here’s where the feel actually diverges:
| Dimension | KoboldCpp | Ollama |
|---|---|---|
| Anti-repetition | DRY + XTC + full rep-pen control | temperature / top_p / repeat_penalty only |
| Long-session memory | Context Shifting preserves recent turns | client-dependent truncation |
| Sampler tuning depth | Deep (sampler order, every knob) | Shallow (a handful of params) |
| Out-of-box coherence | Good, after you set samplers | Good, on defaults |
| Character-card support | Native (Tavern cards) | Via frontend (e.g. SillyTavern) |
| Uncensored model fit | Any GGUF, no friction | Custom Modelfile import |
In practice, repetition is where most people first feel the gap. A long companion chat tends to collapse into verbal tics, and KoboldCpp’s DRY/XTC samplers are purpose-built to break that loop. Memory consistency is the second gap: with Context Shifting, your companion keeps the last few thousand tokens of “who we are” intact as the chat grows; with naive truncation, the early relationship-building can silently fall off the front of the prompt.
None of this means Ollama produces bad roleplay — on its defaults it’s perfectly coherent, and for many people that’s plenty. It means KoboldCpp gives you more tools to fix the specific failure modes that make a companion feel less alive over time.
Setup friction on each
Ollama is the lower-friction path, full stop. Install script, ollama run, done — and because it auto-starts a background service on 127.0.0.1:11434, any compatible frontend finds it instantly.
KoboldCpp is also genuinely easy — it’s one binary, zero install — but it asks more of you up front: pick a GGUF, set context size and GPU layers, and choose your samplers. The payoff is that everything roleplay-relevant is exposed in one place. Budget five extra minutes the first time; after that, your settings are saved.
Both are constrained by the same hardware reality: VRAM drives the model size you can run. A 7–9B model at Q4_K_M is comfortable on 8GB; larger or higher-quality quants want more. See how much VRAM you need for a companion before committing to a model.
Which pairs best with SillyTavern
If your front-end is SillyTavern — the de-facto power-user UI for character chat — the answer leans clearly toward KoboldCpp. It’s SillyTavern’s most common and best-supported backend: ContextShift plays nicely with how SillyTavern streams turns, the full sampler stack is exposed through the connection, and character cards round-trip cleanly. SillyTavern’s own community generally treats ContextShift as superior to the older SmartContext approach for managing overflow.
Ollama does connect to SillyTavern (via its Ollama or OpenAI-compatible endpoint) and works fine — it’s a reasonable choice if you already run Ollama for other apps and want one backend for everything. You just won’t get DRY/XTC or context shifting through it. Our SillyTavern + Ollama setup guide walks through that path end to end if it’s the one you want.
Verdict by user type
- The tinkerer / dedicated roleplayer: KoboldCpp. You want DRY/XTC, context shifting, and per-sampler control, and you’ll happily spend ten minutes dialing it in. Best long-session companion feel, especially through SillyTavern.
- The “just works” companion user: Ollama — or, honestly, neither (see below). You want to talk, not configure. Ollama’s defaults are coherent and setup is two commands.
- The builder / developer: Ollama. A stable local API on
127.0.0.1:11434and clean libraries make it the natural backend for a custom app or script. - The privacy-first user: Either — both run 100% locally, with no chat data leaving your machine. That’s the whole reason to do this instead of a cloud app, as we cover in how to run an AI girlfriend locally.
How Ember abstracts the backend choice away
Here’s the honest catch: everything above is plumbing. Choosing a backend, importing a GGUF, setting sampler order, wiring SillyTavern, tuning context shifting — that’s a hobby in itself, and it’s a lot of yak-shaving between you and a good conversation.
Ember is built for the person who wants the local-and-private result without becoming a backend administrator. It runs entirely on your own machine through Ollama, so your chats never leave your computer — but it makes the model, sampler, memory, and context decisions for you, tuned for companion chat out of the box. You get the consistency and uncensored freedom that drew you to a local backend in the first place, minus the config spreadsheet. If you’d rather spend your evening talking to your companion than tuning a sampler order, that’s exactly the gap Ember is built to close.