Multi-GPU Ollama Setup: Running Bigger Local LLMs

How Ollama Handles Multi-GPU Automatically

Ollama’s own documentation is explicit about the default behavior: “If the model will entirely fit on any single GPU, Ollama will load the model on that GPU. This typically provides the best performance as it reduces the amount of data transferring across the PCI bus during inference. If the model does not fit entirely on one GPU, then it will be spread across all the available GPUs.”

That’s a meaningfully different model from training. Our multi-GPU training setup docs cover DataParallel and DistributedDataParallel, both of which need explicit code changes and, ideally, identical GPUs. Ollama’s inference splitting is automatic and works with whatever’s installed, because it’s solving a different problem: fitting one model’s layers into available memory, not synchronizing gradients across replicas.

Environment Variables That Control It

You don’t need any of these to get basic multi-GPU splitting working, but they matter once you’re running more than one model, sharing the box with a display GPU, or deliberately forcing a spread.

Do You Need NVLink?

No, and this is worth stating plainly because it’s the most common assumption carried over from training setups. Our multi-GPU training guide recommends NVLink where available because training synchronizes gradients across every GPU on every step, and that constant cross-GPU traffic benefits enormously from NVLink’s 10x-plus bandwidth over PCIe.

Inference through Ollama doesn’t work that way. A prompt’s forward pass moves through the model’s layers sequentially: the first group of layers runs on GPU 0, hands its output to GPU 1 for the next group, and so on. That’s a fraction of the data volume and frequency of gradient synchronization. Standard PCIe x8 or x16 is enough; a Gen 4 x8 link and a Gen 3 x16 link both land around 16 GB/s, and either is adequate here.

Can You Mix Different GPUs?

Yes, with a caveat. Ollama’s splitting is based on each GPU’s available memory, not a fixed even divide, so a 24GB card and a 12GB card both get used, just unevenly. That’s a real practical advantage over training, where mismatched GPUs bottleneck the whole job to the slowest card.

The caveat: total usable VRAM is what matters for fitting a model, but the slower or older card in the pair still becomes a throughput bottleneck for whichever layers land on it. Mixing a 4090 with an older 1080 Ti works for loading a bigger model than either card could hold alone, but expect the 1080 Ti’s layers to set the pace for token generation.

Real Hardware Combos for Big Local Models

These are the memory floors from our own hardware guides, mapped to realistic multi-GPU combos.

Full quantization tables and single-GPU options for each model: Qwen3.8-27B, DeepSeek V4 Flash, GLM-5.2.

Common Pitfalls

FAQ

Does Ollama automatically use multiple GPUs?

Yes. Ollama’s own documentation states that if a model fits entirely on one GPU, it loads there for best performance. If it doesn’t fit on any single GPU, Ollama automatically spreads it across all available GPUs. There’s no manual tensor-split flag to configure for basic use.

Do I need NVLink for multi-GPU Ollama?

No. That’s a training requirement, not an inference one. Ollama (via its llama.cpp backend) splits a model’s layers sequentially across GPUs; each token’s forward pass moves through one GPU’s layers, then the next, over standard PCIe. NVLink matters for training, where gradients sync constantly across every GPU. It doesn’t meaningfully speed up Ollama inference.

Can I mix different GPU models or VRAM sizes with Ollama?

Yes, and this is a real advantage over training setups, which need identical GPUs to avoid being bottlenecked by the slowest card. Ollama’s automatic splitting allocates layers based on each GPU’s available memory, so a 24GB card paired with a 12GB card still works, just with an uneven split rather than a clean 50/50.

How do I limit Ollama to specific GPUs?

Set CUDA_VISIBLE_DEVICES to a comma-separated list of GPU indices for NVIDIA cards (ROCR_VISIBLE_DEVICES for AMD, GGML_VK_VISIBLE_DEVICES for Vulkan). This is useful if you have a GPU you want reserved for a display or another workload.

How much combined VRAM do I need to run a 300B-class model like GLM-5.2 or DeepSeek V4 Flash?

It depends on the quantization level. DeepSeek V4 Flash’s smallest usable quant needs about 82.5GB combined, achievable with 4x 24GB GPUs or 2x 48GB cards. GLM-5.2 needs 223GB+ even at its smallest quant, realistically 4x 80GB datacenter GPUs or a very large system RAM offload setup. See our hardware guides for each model’s full quantization table.