If you are a therapist or counselor wondering whether you can use AI without putting your license at risk, the honest answer is that where the model runs matters more than which model it is. Type a client’s presenting problem into a consumer chatbot and you have transmitted Protected Health Information (PHI) to a third party, a HIPAA problem regardless of intent, and regardless of whether a breach ever happens. A local AI runs entirely on your own computer, so the note never leaves the device. This guide covers why cloud AI is a compliance trap for PHI, what “local” actually buys you, realistic clinical use cases, what to run, and the caveats no “AI scribe” vendor will mention.
Is ChatGPT HIPAA compliant? The honest answer
For the version most clinicians actually use, standard ChatGPT, Free or Plus, no. As of mid-2026, OpenAI does not offer a Business Associate Agreement (BAA) for those consumer tiers, and under HIPAA a covered entity may only share PHI with a vendor that has signed one. Pasting anything that could identify a client, symptoms, history, even a distinctive narrative detail, into a no-BAA system is a violation on its face.
There are HIPAA-capable paths at OpenAI: the company signs BAAs for its API and for ChatGPT Enterprise, and through 2026 it has been expanding healthcare-specific offerings and clinician programs. But this is exactly the kind of fast-moving vendor policy you should verify at the source before relying on it, eligibility, tiers, and terms have changed more than once, and a BAA on the API tier is not the same product as the app on your phone.
Here is the part nobody tells you: even a signed BAA does not keep the data on your side of the wire. A BAA is a promise, the vendor agrees not to train on your PHI, to report breaches, and to delete on request, enforced after the fact, once you have already discovered the problem. The PHI still traveled to someone else’s servers, where it can be subpoenaed from the vendor or exposed in the vendor’s breach. The contract narrows the risk; it does not eliminate the transmission. This is the same structural issue we lay out in the AI data privacy guide: once your data is on someone else’s computer, their risk surface becomes yours.
Why local AI sidesteps the whole BAA question
Local AI is different in kind, not degree. When the model weights sit on your workstation and the computation happens on your own GPU, there is no third party in the loop at all. The note is read off your local disk into your local memory and processed there. Nothing is transmitted, so there is no vendor to sign a BAA with, no retention setting to misconfigure, no training pipeline to opt out of, and no server logs to worry about.
That reframes the compliance conversation. The single largest HIPAA risk with AI, disclosing PHI to an outside party, simply does not occur, because there is no outside party. You can even prove it: run the model with a network monitor open and watch nothing leave the machine. Tools like Ollama expose their API only on 127.0.0.1:11434, which is your computer talking to itself. If you want to understand exactly what that local-only guarantee does and does not cover, we go deep in is Ollama really private. This mirrors the calculus we walked through for attorneys in local AI for lawyers, keeping confidential data off the cloud, same problem, different regulated profession.
Cloud AI vs. local AI for PHI
| Cloud AI (no BAA) | Cloud AI (with BAA) | Local AI | |
|---|---|---|---|
| Where PHI goes | Vendor’s servers | Vendor’s servers | Stays on your machine |
| BAA required for PHI | Yes, and absent | Yes, signed | None needed (no third party) |
| Third-party breach exposure | High | Reduced, not zero | None |
| Works fully offline | No | No | Yes |
| Protection type | , | Contractual promise | Architectural |
| Ongoing cost | Subscription | Enterprise pricing | Hardware you already own |
The middle column is the trap most guides stop at: “get a BAA and you’re fine.” A BAA beats nothing, but it is a contract about behavior. The right column is a property of the system.
Realistic use cases (clinician always reviews)
Local models in the 8B-32B range are not frontier systems, but for document-grounded, structured tasks they are genuinely useful today:
- Progress and SOAP notes from your own shorthand. You jot the clinical facts; the model expands them into a formatted draft. You then correct and sign it. The model never sees more than you deliberately give it.
- Treatment-plan drafting. Ask for goal/objective/intervention scaffolds for a presentation, then apply your clinical judgment. It is a starting outline, not a plan.
- Psychoeducation handouts. Plain-language explainers on anxiety, sleep hygiene, or a coping skill, generic content with no client data in it, which is the safest use of the lot.
- Summarizing your own material. Condense a long intake into a one-paragraph summary, or pull a timeline out of your own session notes.
The non-negotiable rule: the AI is an assistant that never gets the final word. Local solves confidentiality, not accuracy, a model on your laptop invents details exactly as readily as one in the cloud. Every note it drafts is your note, reviewed and corrected by you before it enters the record. For an upload-and-ask workflow over your own documents, see chat with documents locally; for a sense of what these models handle well versus poorly, best private AI for sensitive questions is a good gut check.
What to run: Ollama plus a capable local model
You do not need to be an engineer. The engine is Ollama, a one-line install that runs the model and exposes a local-only API. Our how to install Ollama walkthrough covers every OS, but the short version is:
curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen3:14bFor note drafting and summarizing, a mid-size instruction-following model is the sweet spot. Real sizing numbers, no hand-waving:
| Model class (2026) | Footprint (Q4_K_M) | Practical VRAM | Good for |
|---|---|---|---|
| 8B (e.g. Qwen3 8B) | ~5-6 GB | 8-12 GB GPU or 16 GB Mac | Fast note drafts, handouts |
| 12-14B (Gemma 3 12B, Qwen3 14B) | ~8-10 GB | 12-16 GB GPU | Notes, summaries, plans |
| 24-32B (Mistral Small 3.2 24B, Qwen3 32B) | ~14-20 GB | 24 GB GPU | Longer notes, more nuance |
Q4_K_M is a quantization level, it compresses the weights so the model fits in consumer memory with little quality loss. A single mid-range GPU or a 16-32 GB Apple Silicon Mac covers the 8B-14B tier comfortably, which is plenty for note work. Start there; step up to 24B-32B only if you want more polish. The table lists proven, current picks; the newest generations, Gemma 4 (a 12B unified multimodal model) and Qwen3.6, are drop-in upgrades at roughly the same footprints if you want the latest, but nothing in this tier is a wrong answer for note drafting.
The caveats no AI scribe vendor will tell you
Local AI removes the transmission risk. It does not remove your responsibilities, and it adds a couple:
- You are still the responsible party. The clinical record is yours. The model drafts; you verify, correct, and sign every word.
- De-identify where practical anyway. Feeding shorthand with initials instead of full names costs you nothing and is good hygiene, even offline.
- Now the security is on your device. Full-disk encryption, a strong login, auto-lock, and physical security become your HIPAA safeguards. Never expose the Ollama API to the internet, keep it on
127.0.0.1. - Check your board and jurisdiction. State licensing rules, your malpractice carrier, and (where applicable) informed-consent norms around AI use vary. Some settings expect disclosure to clients.
- This is not legal or compliance advice. It is a technical guide to keeping PHI off the cloud. Confirm your obligations with your licensing board, your practice’s compliance officer, or a healthcare attorney before using any tool on real client data.
The bottom line: for PHI, architecture beats contracts. Running the model yourself is the only setup where “the data never left” is a fact about the system, not a promise from a vendor.
Own your private AI
Here is the uncomfortable truth about “turnkey”: for PHI, there is no hosted shortcut. Any product that runs on someone else’s servers, however well-behaved, puts you right back in BAA territory, which is the problem this whole article exists to avoid. The private-by-construction setup is the stack above: Ollama on your own machine, a mid-size model sized to your hardware, the API kept on 127.0.0.1, and you reviewing every note. It takes an afternoon, not an engineering degree, how to install Ollama and chat with documents locally walk you through it end to end. Build it once, keep yourself in the loop on every note, and you get real AI help without handing your clients’ most sensitive information to someone else’s servers.