If you’ve maxed out a single 24GB card and keep bumping into the ceiling, 70B models that won’t fit, context windows that spill into system RAM and crawl, the obvious next move is a second GPU. The good news: a dual GPU local LLM setup genuinely works, and it’s the cheapest realistic path to running 70B-class models entirely on hardware you own. The important caveat: two GPUs don’t merge into one big GPU. They don’t truly pool VRAM into a single shared pool the way marketing might suggest. What actually happens is that the inference engine splits the model across both cards, and that distinction explains almost everything about what helps, what’s a myth, and what you can skip buying.
This guide walks through how the split actually works, whether you need NVLink (you almost certainly don’t for inference), the PCIe-lane question everyone overthinks, mixing different cards, the power and cooling reality, and exactly how Ollama and llama.cpp handle the whole thing.
What “Pooling VRAM” Really Means
Here’s the mental model that matters. When you run a model across two GPUs, the engine doesn’t fuse 24GB + 24GB into a magic 48GB framebuffer. Instead it partitions the model, most commonly by layer, and puts different chunks on different cards. Card A holds the first half of the layers; card B holds the second half. During generation, the activation for a token flows through card A’s layers, gets handed across the PCIe bus to card B, flows through the rest, and produces the output.
The practical upshot is the part people care about: you can load a model that no single card could hold. A 70B model at roughly 4-bit quantization needs somewhere in the ballpark of 40GB just for weights, plus headroom for context. That doesn’t fit in 24GB, and offloading the overflow to system RAM tanks your speed. Put two 24GB cards together and the engine sees ~48GB of combined capacity, enough to hold the whole thing in VRAM with room for a real context window. If you want the full breakdown of the numbers, our guide on how much VRAM you actually need for a 70B model lays out the weight-plus-context math.
So “pooling” is a useful shorthand for the outcome (bigger models fit), but the mechanism is splitting. Keeping that straight is what lets you correctly answer the next three questions.
Do You Need NVLink? (Almost Certainly Not)
NVLink, NVIDIA’s high-bandwidth direct GPU-to-GPU bridge, is the single most over-recommended purchase in home AI. For inference, you generally don’t need it.
The reason traces straight back to the split model above. In the common layer-split mode, the only data that crosses between cards is the activation tensor handed off at the boundary between one card’s layers and the next. That’s a small payload relative to the giant matrix multiplications happening inside each card, and it only crosses the bus a couple of times per token. The PCIe bus handles it comfortably. NVLink does offer dramatically higher card-to-card bandwidth, but inference simply doesn’t generate enough cross-card traffic for that headroom to pay off in most setups.
Where NVLink earns its keep is training and heavy tensor-parallel / row-split workloads, where the cards must constantly exchange large gradient or partial-result tensors. That’s a different job. If your goal is to run models, not train them, skip NVLink and put the money toward VRAM or a better PSU. (It’s also worth noting NVLink is restricted to specific cards and bridge spacings, plenty of popular setups like dual 3090s technically support it, but the point stands: for inference you won’t miss it.)
PCIe Lanes: x8/x8 Is Fine
The follow-up worry is PCIe bandwidth. Most consumer motherboards, when you populate two x16 slots, silently drop both to x8/x8, because mainstream CPUs only expose so many lanes. People panic about this. They mostly shouldn’t.
For the same reason NVLink is overkill, x8 per card is enough for inference. The cross-card handoff in layer-split mode is light enough that the difference between x16 and x8 is, in practice, small to negligible for generation speed. PCIe 4.0 x8 (or even 3.0 x8 on older boards, with a slightly larger but still modest penalty) keeps two cards fed fine.
Two honest caveats:
- Model load time scales with bus speed, a wider, faster link gets the weights onto the cards quicker. That’s a one-time cost per load, not a per-token cost.
- Tensor-parallel (row-split) modes that try to split individual weight matrices across cards are bandwidth-hungry and will feel x8 more. Most home users stick with layer split, where it doesn’t bite.
Don’t go out and buy a Threadripper/EPYC platform for the lane count just to run inference. A normal consumer board running x8/x8 is a perfectly good dual GPU local LLM host.
Can You Mix Cards? (Yes, With Caveats)
You don’t need two identical GPUs. The engine can split across mismatched cards, say a 3090 paired with a 3060, or two different 24GB models, and it will weight the split toward each card’s capacity. This is great for upgrading incrementally: keep your existing card, add a second.
The realities to accept when mixing:
- VRAM is what you’re buying. Two 24GB cards (e.g. 2×3090) give you ~48GB to work with, that’s the configuration that cleanly unlocks 70B. Pairing a 24GB card with a 12GB card gives you ~36GB, which helps but won’t comfortably hold a 70B at a good quant.
- The slower card can gate throughput. Since the token flows through both cards in sequence, a much weaker second card can become the bottleneck for that portion of the model. Mixing a fast and a slow card works, but don’t expect the fast card’s solo speed.
- Generation mismatches are usually fine; architecture quirks vary. Sticking within one or two GPU generations keeps drivers and behavior predictable.
For most people the sweet-spot pairing is two of the same proven 24GB card. The used 3090 remains the value king here, two of them is the canonical home 70B rig, and our take on why the used 3090 is still the best value for local AI explains the $/GB-VRAM math that makes a dual-3090 box so popular.
How Ollama Handles Multi-GPU
The best part of a modern two-card setup: Ollama does this automatically. Recent Ollama builds detect all your GPUs at startup and distribute the model’s layers across them with no configuration required. You don’t edit a config file, you don’t pass a flag, you just ollama pull a model that’s too big for one card, run it, and Ollama spreads it across both.
ollama pull llama3.3:70b
ollama run llama3.3:70bIf the model fits in your combined VRAM, it loads across the cards and runs. You can sanity-check what’s loaded and where with:
ollama psA couple of practical notes for multi-GPU Ollama:
- It distributes by layer, the compatible default, which is exactly what you want for a PCIe-connected pair.
- If you ever need to restrict which cards Ollama uses (say you want to keep one GPU free for display or another task), the
CUDA_VISIBLE_DEVICESenvironment variable controls which GPUs are visible to it. - Make sure the model genuinely fits. If combined VRAM is short, Ollama will offload the remainder to system RAM, and you’ll watch speed collapse, the same spill problem a single card hits, just at a higher ceiling. The line between VRAM and RAM is the whole game here, and why VRAM matters far more than system RAM is worth reading before you size a model to your pair.
How llama.cpp Handles Multi-GPU
If you run llama.cpp directly (or a wrapper around it), you get more explicit control. Two flags matter:
--split-mode(-sm) chooses how the model is split. The options are roughly:layer, the default and recommended mode: splits the model by layer across cards. Best compatibility, friendliest to PCIe dual-card systems. Start here.row, a tensor-parallel-style path that splits weight matrices across cards. Current llama.cpp marks it deprecated/slow (a newer, still-experimentaltensormode supersedes it for true tensor parallelism); like any cross-card weight split it’s bandwidth-hungry, so it’s only worth bothering with on high-bandwidth interconnects, not a PCIe home pair.none, keep the whole model on one GPU.
--tensor-split(-ts) sets the ratio of how much goes to each card. For two equal cards you can leave it default; for mismatched cards you can bias it, e.g.-ts 24,12to weight the split roughly toward each card’s capacity.
A minimal two-GPU launch looks like:
llama-server -m model-70b-Q4_K_M.gguf -ngl 99 -sm layer-ngl 99 (n-gpu-layers) tells it to put as many layers on GPU as possible; with two cards and -sm layer, those layers spread across both. If you’re tuning a mismatched pair, add -ts to shape the balance. The honest default for a home rig is simple: layer split, default ratio on matched cards, biased ratio on mismatched ones.
The Numbers: What Two Cards Actually Get You
Treat all of these as ranges, not promises, real throughput depends on the exact model, quant, context length, cooling, and software version.
| Setup | Combined VRAM | Largest comfortable model | Rough 70B Q4 speed | Notes |
|---|---|---|---|---|
| Single 24GB card | ~24GB | 32B at Q4 (70B only with slow offload) | N/A on-GPU | 70B spills to RAM, drags |
| 2×24GB (e.g. 2×3090) | ~48GB | 70B at Q4 with real context | Roughly 15-25 tok/s, ballpark | The canonical home 70B rig |
| 24GB + 12GB mismatched | ~36GB | Large 30B-class comfortably | Varies, gated by slower card | Good upgrade path, not ideal for 70B |
Two things to internalize about scaling. First, two cards do not double your speed. A model split across cards runs through them in sequence, plus a little bus overhead, so you’re capturing well under a full 2×, think a meaningful boost in capacity rather than a doubling of throughput. Second, the speed that matters is whether generation keeps up with your reading pace. A 70B on a dual-3090 box in the rough 15-25 tok/s band is comfortable for interactive chat, responses stream smoothly rather than blazing, which is exactly the bar our piece on what tokens-per-second is actually usable argues you should aim for. For an uncensored 70B specifically, where the bigger model is noticeably more coherent and less repetitive, the case for two cards gets stronger; see GPU choices for running an uncensored 70B at home.
Power, PSU, and Cooling, the Unsexy Part
This is where dual-GPU builds actually go wrong, so plan it before you buy the second card.
- PSU headroom. Two big cards draw real power. A single high-end 24GB card can pull in the 300-350W neighborhood at stock, so two of them plus a CPU pushes total system draw well up. A quality 1000W-1200W unit is the sane range for a dual-3090-class build; don’t run a PSU near its limit, because these cards have real transient spikes. (Exact figures vary by card and load, size for headroom, not the average.)
- Undervolt for an always-on box. For inference you don’t need the top of the voltage-frequency curve. Power-limiting or undervolting each card typically costs only a small single-digit percentage of speed while shedding a lot of heat, noise, and electricity cost, the right default for a 24/7 local-AI machine.
- Physical space and airflow. Two triple-slot cards crammed together is the classic mistake: the top card starves the bottom one of intake and both cook. Look for a case with real spacing, consider a board layout that separates the slots, and make sure front-to-back airflow is genuinely there. Some people use a riser to relocate the second card.
If you’re speccing the whole machine from scratch rather than just adding a card, our best budget AI PC build covers the PSU, case, and cooling choices that keep a multi-GPU box stable.
The Honest Bottom Line
A dual GPU setup is the most accessible way to run 70B-class models entirely at home, but be clear-eyed about why it works. You’re not building a single bigger GPU; you’re letting the engine split one model across two cards. That framing tells you the truth on every adjacent question: NVLink is unnecessary for inference, x8/x8 PCIe is fine, mixed cards work but the slower one can gate you, and the win is capacity, not a doubling of speed. Software-wise it’s genuinely easy now, Ollama auto-distributes layers with zero config, and llama.cpp gives you -sm layer plus -ts when you want to tune.
Spend your money where it matters: total VRAM, a PSU with real headroom, and a case that can actually cool two cards. Skip the NVLink bridge and the exotic high-lane platform. Do that, and a two-card box will hold a 70B model in VRAM and chat back at a comfortable pace, fully private, fully yours, with nothing leaving the machine.