AirLLM: Run a 70B LLM on a Single 4GB GPU (How It Works, and the Catch)

How AirLLM Actually Works

Every other way to fit a big model into a small GPU changes the model: quantization shrinks the weights, pruning removes parameters, distillation trains a smaller model to imitate the big one. AirLLM changes none of that. The 70B model stays exactly as trained, at full precision if you want it. What changes is that it’s never fully in VRAM at the same time.

A transformer is just a stack of near-identical layers run in sequence. AirLLM splits the model into those layers on disk, then during inference it loads layer 1 into VRAM, runs the forward pass through it, frees that VRAM, loads layer 2, runs it, frees it, and so on through the whole stack. At any given moment, VRAM only has to hold one layer’s worth of weights, not the whole model. That’s how a model with 140GB of FP16 weights can run on a 4GB card: you never need more than a few hundred MB in VRAM at once.

What Model Sizes Fit at What VRAM

These are the numbers from the project’s own documentation, current as of its 2026 releases:

The Speed Tradeoff Nobody Puts in the Headline

Here’s the part that matters more than the “70B on 4GB” headline: reading model weights from disk on every forward pass is fundamentally slower than reading them from VRAM. GDDR7 on an RTX 5090 moves data at 1,792 GB/s. Even a fast PCIe 5.0 NVMe tops out around 14 GB/s sequential, and inference read patterns aren’t always sequential. That gap, over 100x, doesn’t disappear because the model technically runs.

In practice this means AirLLM is not something you’d use for interactive chat. It’s a way to run a forward pass on a model you otherwise have zero access to on your hardware, for evaluation, one-off inference, testing whether a model’s outputs are even worth pursuing further, or research on a budget. The project does offer optional block-wise 4-bit/8-bit compression that the maintainer reports gives up to a 3x speedup with close to no accuracy loss, and prefetching to overlap loading with compute, both of which help, but neither changes the fundamental disk-bound nature of the technique.

When AirLLM Makes Sense vs. the Alternatives

Getting Started

Installation is a single pip package:

pip install airllm

Then point it at any supported Hugging Face repo ID:

from airllm import AutoModel

model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

input_text = ["What is the capital of United States?"]
input_tokens = model.tokenizer(input_text, return_tensors="pt",
    return_attention_mask=False, truncation=True, max_length=128, padding=False)

generation_output = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=20,
    use_cache=True,
    return_dict_in_generate=True)

output = model.tokenizer.decode(generation_output.sequences[0])
print(output)

On first run, AirLLM downloads the model and splits it into layer-wise shards on disk, so budget disk space roughly equal to the model’s full size before it starts serving requests. The project is Apache 2.0 licensed and actively maintained, with MoE and FP8 support added as recently as mid-2026.

Frequently Asked Questions