You want to ask questions over your own files, contracts, research papers, years of notes, the way people interrogate ChatGPT, but without uploading a single page to anyone’s servers. AnythingLLM is the cleanest way to get there. It’s a free, open-source desktop app that wraps a complete retrieval pipeline, document ingestion, embeddings, a vector database, and a chat model, behind a normal-looking chat window. Point it at a local engine like Ollama, drop in your documents, and you have a private “ChatGPT for your own files” that runs entirely on hardware you own.
This guide walks the whole AnythingLLM setup end to end: installing it, connecting it to a local model, configuring the embedder (the step most people get wrong), organizing files into workspaces, and confirming that nothing leaves your machine. No account, no API key, no monthly bill required when you run it fully local.
What AnythingLLM actually is
Strip away the marketing and AnythingLLM is a local RAG app, a front end and orchestration layer for retrieval-augmented generation. It doesn’t ship its own large language model brain that you’re locked into; instead it acts as a bridge to whatever LLM provider you choose. That provider can be a cloud API (OpenAI, Anthropic) or, the whole reason you’re here, a local engine running on your own box.
There are two ways to run it:
- The desktop app (macOS, Windows, Linux), a single-user install with a built-in LLM engine, a built-in CPU-based embedder, and a bundled vector database. One-click setup, no account, nothing to configure if you just want it working in five minutes.
- The self-hosted Docker version, multi-user, with workspace sharing and admin controls, meant for teams or a home server.
For “chat with my own documents on my own laptop,” the desktop app is what you want. The killer feature for privacy-minded readers: by default, the model, your documents, your embeddings, and your chats are all stored locally on your machine. AnythingLLM describes itself as private by default, and when you keep every component local, that’s literally true, there’s no remote subprocessor in the loop to trust.
Why pair AnythingLLM with Ollama
The desktop app’s built-in engine is genuinely convenient, it lets a non-technical person open the app and start chatting with zero setup. But most people who care about ownership end up pairing AnythingLLM with Ollama instead, for a few honest reasons:
- Model choice. Ollama gives you a huge catalog you can pull with one command and swap freely, Llama, Qwen, Mistral, Gemma, and uncensored variants. The built-in engine is more of a get-started default.
- GPU acceleration. Ollama uses your GPU properly. The desktop app’s bundled embedder runs on CPU, which is fine for indexing but slower on big batches.
- One engine, many apps. If you already run Ollama, the same local model can serve AnythingLLM, a chat UI like Open WebUI, and your own scripts, all off the same loopback endpoint.
If Ollama isn’t installed yet, start with our how to install Ollama walkthrough, it’s a one-line install on every platform. If you’re still deciding which local runtime to commit to, Ollama vs LM Studio vs Jan compares the realistic options; AnythingLLM can talk to all three, so you’re not locked in either way.
Pull a chat model and an embedding model before you open AnythingLLM:
ollama pull llama3.1:8b
ollama pull nomic-embed-textThe 8B chat model is a sensible starting generator; size it to your hardware. If you’re unsure what fits, the best local LLM for 12-16GB VRAM breaks down what runs comfortably on common cards. nomic-embed-text is the embedding model, more on why you need a separate one in a moment. It’s small (around 270MB) and adds very little memory overhead.
Step 1: install AnythingLLM and pick your LLM provider
Download the desktop app from the official AnythingLLM site for your OS and install it like any normal application. On first launch it runs a short onboarding wizard that asks two key questions: which LLM provider to use, and which embedder.
For the LLM provider, choose Ollama. AnythingLLM auto-detects the default Ollama endpoint at http://127.0.0.1:11434, the same loopback address everything local runs on. If detection works, your pulled models appear in a dropdown; pick the chat model you pulled (e.g. llama3.1:8b). If the list is empty, make sure Ollama is actually running (ollama list in a terminal should show your models) and that the URL points at 127.0.0.1:11434.
That 127.0.0.1 matters: it’s a loopback address that never touches the network. Traffic to it physically can’t leave your computer. If you want to understand exactly how airtight that is, and the one or two settings that could leak telemetry if you’re careless, see is Ollama really private.
Step 2: configure the embedder (don’t skip this)
This is the step people quietly get wrong, then wonder why document chat feels dumb. Document chat needs two models, not one:
- A chat model that reads retrieved passages and writes the answer (your Llama/Qwen/etc.).
- An embedding model that turns your documents, and each question, into vectors so the app can find the relevant passages in the first place.
AnythingLLM ships with a built-in CPU embedder, so it works out of the box even if you change nothing. That’s the right call if you want zero friction. But if you’ve gone to the trouble of running Ollama, point the embedder at Ollama too and select nomic-embed-text. Keeping both models on the same local engine means your document text gets embedded locally as well, no text is sent anywhere to be vectorized.
| Role | What it does | Local choice |
|---|---|---|
| LLM provider | Writes the actual answer from retrieved context | Ollama → llama3.1:8b (or any pulled chat model) |
| Embedder | Converts documents + queries into searchable vectors | Built-in (default) or Ollama → nomic-embed-text |
| Vector database | Stores vectors and finds the nearest matches | LanceDB, built in, no setup |
One caution worth stating plainly: if you ever re-point the embedder at a different model after you’ve already indexed documents, you’ll generally need to re-embed those documents. Query vectors and stored vectors have to come from the same embedding model to be comparable. Decide on your embedder before you load a big corpus, and you’ll avoid re-indexing.
Step 3: the local vector database
By default AnythingLLM uses LanceDB, an embedded, serverless vector database that runs inside the app itself, no separate database server to install or manage. Your document text and embeddings stay on your disk, and per AnythingLLM’s own documentation, the vectors never leave the application when you use this default.
You can configure external vector databases (Chroma, Pinecone, Qdrant, Weaviate and others are supported), but for a private local setup there’s no reason to. The built-in LanceDB is the option that keeps everything on one machine. If you’re curious how this same three-part stack, embedder, vector store, generator, works under the hood, our build a local RAG with Ollama guide shows the from-scratch version in about 40 lines of Python. AnythingLLM is that exact pipeline with the wiring done for you.
Step 4: create a workspace and add documents
AnythingLLM organizes everything into workspaces, think of them as separate rooms, each with its own documents, its own conversation history, and (optionally) its own model settings. This is more useful than it first sounds:
- A “Contracts” workspace holds only legal documents, so answers can’t bleed in from unrelated files.
- A “Research” workspace indexes your papers and notes.
- A “Personal” workspace stays walled off from work material.
Workspaces keep retrieval focused. A question asked in one workspace only searches that workspace’s documents, which makes answers more relevant and easier to trust. You can even assign different chat models per workspace, a small fast model for quick lookups, a larger one for harder reasoning.
To load files, open a workspace, click the upload/manage-documents control, and drag in your files. AnythingLLM handles common document formats, PDFs, Word documents, plain text, and markdown among them, and also offers data connectors and a website scraper for pulling in web pages (the exact supported list lives in the app’s upload dialog, so check there for anything unusual). After uploading, you move or embed the documents into the workspace; that’s the step that chunks them, runs them through your embedder, and writes the vectors into LanceDB. Larger or scanned PDFs take longer, and a scanned image-only PDF with no text layer won’t extract well, that’s an OCR limitation, not an AnythingLLM bug.
Once embedding finishes, just ask questions in the chat box. The app retrieves the most relevant chunks, hands them to your local model, and writes a grounded answer. Many setups can show or cite the source passages, which is the whole point of RAG over a bare chatbot, you can check the answer against the actual text instead of trusting a confident guess.
Step 5: prove nothing is leaving your machine
This is the part that matters for sensitive files, and it’s easy to verify. With AnythingLLM configured fully local, Ollama as both the LLM and the embedder, LanceDB as the vector store, disconnect from the internet entirely and ask a question. It still works. Indexing still works. That’s the strongest possible proof that the whole pipeline lives on 127.0.0.1 and nothing is phoning home.
Two honest caveats so you’re not surprised:
- Telemetry. AnythingLLM has historically included anonymous, opt-out usage telemetry (no document content). If you want zero outbound chatter, find the telemetry toggle in settings and turn it off, the airplane-mode test above will confirm there’s nothing left talking.
- Provider choice is on you. AnythingLLM will happily connect to OpenAI or Anthropic if you select them. The privacy guarantee only holds when every component, LLM, embedder, vector DB, is set to a local option. The app makes that easy, but it doesn’t enforce it.
Get those right and you have what cloud doc-chat can only promise: not “we don’t train on your data,” but no transmission in the first place. For professionals who legally can’t paste client files into a public chatbot, that distinction is everything, see local AI for lawyers and confidential work for why “the file never left the building” beats any cloud privacy policy, and the private ChatGPT alternative that keeps your data for the broader case.
AnythingLLM vs. the alternatives
AnythingLLM isn’t the only way to chat with documents locally, and it’s worth being honest about where it sits:
- Versus a DIY Python RAG: AnythingLLM trades control for convenience. You don’t get to tune chunk sizes and retrieval logic as freely, but you also don’t write or maintain any code. If you want that control, build it yourself; if you want results today, use the app.
- Versus Open WebUI: Open WebUI is a superb general chat front end for Ollama with document features bolted on; AnythingLLM is RAG-first, with workspaces and document management as the main event. Many people run both for different jobs.
- Versus the built-in engine alone: Skipping Ollama and using AnythingLLM’s bundled engine is the fastest possible start and totally valid for a casual user. You just give up model variety and GPU speed.
The bottom line
AnythingLLM is the shortest honest path from “I have a folder of files” to “I can ask them anything, privately.” Install the desktop app, point both the LLM and the embedder at Ollama, let LanceDB hold your vectors, organize files into workspaces, and run the airplane-mode test to prove it’s sealed. The result is a fully-local, no-subscription, no-account assistant that reads your documents and answers from them, with your data staying exactly where it started, on your own machine.
The setup takes one afternoon and pays off every time you’d otherwise have pasted something sensitive into a box you don’t control.