Buying guides tell you which GPU to buy. This post tells you why, with training throughput numbers, inference speeds, and VRAM requirements across every major card in 2026. Whether you are choosing between an RTX 5090 and an H100 rental or deciding if an RTX 5060 Ti is enough for your use case, the numbers below will guide that decision.
Quick Navigation:
Benchmark Methodology
All relative performance numbers use PyTorch 2.x with mixed precision (BF16/FP16) unless noted. The RTX 5090 is set as the 100-point baseline for consumer card comparisons. Professional cards (A100, H100) are included in a separate column since they operate in a different cost bracket.
Key metrics tracked:
- Training throughput: tokens/second or images/second at a fixed batch size
- Inference speed: tokens/second for autoregressive generation
- Memory bandwidth: the true bottleneck for most modern workloads
- VRAM ceiling: what models fit without CPU offloading
Why relative numbers: Absolute benchmark figures shift with driver updates, framework versions, and batch sizes. Relative performance between cards is far more stable and more useful for buying decisions.
GPU Performance Ladder
| GPU | VRAM | Bandwidth | FP16 Tensor TFLOPS | Training Score | Inference Score | Price Tier |
|---|---|---|---|---|---|---|
| H100 SXM5 80GB | 80 GB HBM3 | 3,350 GB/s | 989 | 350 | 187 | Cloud only (~$2.49/hr) |
| A100 SXM4 80GB | 80 GB HBM2e | 2,000 GB/s | 312 | 180 | 112 | Cloud only (~$1.99/hr) |
| RTX 5090 32GB | 32 GB GDDR7 | 1,792 GB/s | 838 | 100 | 100 | ~$2,200-2,700 |
| RTX 5080 16GB | 16 GB GDDR7 | 960 GB/s | 670 | 55 | 54 | ~$1,000-1,200 |
| RTX 5070 Ti 16GB | 16 GB GDDR7 | 896 GB/s | 437 | 42 | 50 | ~$750-900 |
| RTX 5060 Ti 16GB | 16 GB GDDR7 | 448 GB/s | 280 | 25 | 25 | ~$450-550 |
| RTX 3060 12GB | 12 GB GDDR6 | 360 GB/s | 101 | 14 | 20 | ~$230-280 |
Reading the table: Training score is most relevant for model fine-tuning and pretraining. Inference score matters for local LLM serving. Note that the RTX 5070 Ti scores higher on inference than training relative to the 5080, because inference is more bandwidth-bound and both cards share similar bandwidth despite the 5080 having more compute cores.
LLM Training Benchmarks
Training large language models is the most demanding deep learning workload. It requires both high VRAM capacity and high memory bandwidth, since gradient updates touch every parameter on every step.
Transformer Training Throughput (BF16, PyTorch 2.x)
| GPU | GPT-2 (117M) tok/s | Llama 7B (LoRA) tok/s | Llama 13B (LoRA) tok/s | Notes |
|---|---|---|---|---|
| H100 SXM5 | ~95,000 | ~6,500 | ~3,200 | NVLink + HBM3 advantage |
| A100 SXM4 | ~50,000 | ~3,300 | ~1,600 | HBM2e, ECC memory |
| RTX 5090 | ~27,000 | ~1,850 | ~820 | Best consumer card |
| RTX 5080 | ~15,000 | ~1,020 | ~450 | VRAM limit at 13B+ |
| RTX 5070 Ti | ~11,500 | ~780 | ~340 | Good value for 7B work |
| RTX 5060 Ti | ~6,700 | ~460 | ~200 | Learning/prototyping |
| RTX 3060 12GB | ~3,800 | ~260 | Does not fit | Budget starting point |
VRAM Requirements by Model Size
This is the hard constraint that determines whether a GPU can even run your workload.
| Model Size | Full Precision (FP16) | QLoRA Fine-tune | Minimum GPU |
|---|---|---|---|
| 7B parameters | ~14 GB | ~6 GB | RTX 3060 12GB (QLoRA) |
| 13B parameters | ~26 GB | ~10 GB | RTX 5060 Ti 16GB (QLoRA) |
| 30B parameters | ~60 GB | ~20 GB | RTX 5090 32GB (QLoRA), A100 (FP16) |
| 70B parameters | ~140 GB | ~48 GB | 2x A100 (FP16), H100 (QLoRA) |
| 405B parameters | ~810 GB | ~270 GB | Multi-node H100 cluster |
QLoRA changes the game: Techniques like QLoRA (4-bit quantization with low-rank adapters) reduce VRAM requirements dramatically. You can fine-tune a 13B model on an RTX 5060 Ti 16GB that would otherwise need 26 GB. For most practical fine-tuning work, VRAM requirements in the QLoRA column apply.
Local LLM Inference Benchmarks
Inference is bandwidth-bound. The GPU that moves weights from VRAM to the compute units fastest wins. This is why the H100 dominates inference despite its professional price tag.
| GPU | Llama 8B Q4 (tok/s) | Llama 70B Q4 (tok/s) | Qwen 32B Q4 (tok/s) | Notes |
|---|---|---|---|---|
| H100 SXM5 80GB | ~350 | ~95 | ~165 | All models fit fully in VRAM |
| A100 80GB | ~210 | ~58 | ~98 | All models fit fully in VRAM |
| RTX 5090 32GB | ~188 | ~52 (partial offload) | ~70 | Qwen 32B Q4 fits (~19 GB) |
| RTX 5080 16GB | ~100 | ~12 (heavy offload) | ~22 (partial offload) | Limited for 30B+ models |
| RTX 5070 Ti 16GB | ~94 | ~11 (heavy offload) | ~20 (partial offload) | Similar bandwidth to 5080 |
| RTX 5060 Ti 16GB | ~47 | ~8 (heavy offload) | ~12 (partial offload) | Usable for 7B-13B models |
| RTX 3060 12GB | ~38 | ~5 (heavy offload) | ~7 (heavy offload) | Budget entry point |
Offloading penalty: When a model does not fit in VRAM, layers are offloaded to system RAM and loaded on demand over the PCIe bus. PCIe 5.0 maxes at around 64 GB/s vs GDDR7’s 1,792 GB/s on the RTX 5090. Models that partially offload to RAM run at a fraction of full-VRAM speed. If you want to run 70B models locally at usable speeds, you need a professional card or a dual-GPU setup.
Image Generation Benchmarks
Image generation with FLUX and Stable Diffusion is a mix of compute and bandwidth. Higher VRAM enables larger batch sizes and higher resolution without tiling.
| GPU | FLUX Dev FP8 (1024px, it/s) | SDXL FP16 (1024px, it/s) | FLUX Dev FP16 (1024px) |
|---|---|---|---|
| RTX 5090 32GB | ~4.8 it/s | ~12 it/s | ~2.3 it/s (fits fully) |
| RTX 5080 16GB | ~3.2 it/s | ~9 it/s | Requires tiling or CPU offload |
| RTX 5070 Ti 16GB | ~2.6 it/s | ~7.5 it/s | Requires tiling or CPU offload |
| RTX 5060 Ti 16GB | ~1.6 it/s | ~4.5 it/s | Requires tiling or CPU offload |
| RTX 3060 12GB | ~0.8 it/s | ~2.5 it/s | Heavy tiling required |
For image generation, the RTX 5080 hits a practical sweet spot. FLUX Dev FP8 runs well within its 16 GB, and SDXL is fast. The 5090’s advantage for image gen is running FLUX Dev FP16 natively without memory tricks, which benefits professional workflows generating hundreds of images.
VRAM Requirements by Workload
12 GB VRAM (RTX 3060)
Good for: fine-tuning 7B models with QLoRA, running 7B models at full speed, SDXL image generation, computer vision training on standard datasets.
Not good for: anything 13B or larger without heavy quantization, FLUX Dev in full precision, multi-model inference pipelines.
16 GB VRAM (RTX 5060 Ti, 5070 Ti, 5080)
Good for: fine-tuning 13B models with QLoRA, running 13B models comfortably, FLUX Dev FP8, most computer vision and NLP research workloads.
Not good for: 30B+ models without offloading, FLUX Dev FP16, full-precision 13B fine-tuning.
32 GB VRAM (RTX 5090)
Good for: fine-tuning up to 30B models with QLoRA, running Qwen 32B Q4 fully in VRAM, FLUX Dev FP16 natively, multi-model inference setups, serious research workflows.
Not good for: 70B+ models at full precision, anything requiring NVLink between multiple GPUs for VRAM pooling.
80 GB VRAM (A100, H100)
Good for: pretraining and fine-tuning 70B models with full precision, production inference at scale, multi-tenant GPU serving. This is the professional tier where the hardware is rented, not owned.
Pair with NVLink for 160 GB combined VRAM in dual configurations.
Price/Performance Analysis
Raw performance is only half the story. For most users, price per token/second or price per training step matters more than absolute throughput.
| GPU | Approx. Price | Training Score | Perf per $1,000 | Verdict |
|---|---|---|---|---|
| RTX 3060 12GB | ~$250 | 14 | 56 | Best value for learning |
| RTX 5060 Ti 16GB | ~$500 | 25 | 50 | Best entry 16 GB option |
| RTX 5070 Ti 16GB | ~$825 | 42 | 51 | Strong price/performance |
| RTX 5080 16GB | ~$1,100 | 55 | 50 | Similar ratio to 5070 Ti |
| RTX 5090 32GB | ~$2,450 | 100 | 41 | Lower ratio but more VRAM |
The VRAM premium: The RTX 5090 has the lowest performance-per-dollar of any consumer card above, but it is the only consumer option that handles 30B models and FLUX FP16. You are paying a premium not just for speed but for VRAM capacity. If your workload fits in 16 GB, the RTX 5070 Ti offers nearly identical price/performance to the 5090 at less than half the price.
Recommendations by Use Case
Learning and Coursework
RTX 3060 12GB or RTX 5060 Ti 16GB. The 3060 handles every PyTorch tutorial and Hugging Face course without issue. The 5060 Ti 16GB adds headroom for 13B models if you want to experiment with local LLMs too.
Research and 7B-13B Fine-tuning
RTX 5070 Ti 16GB or RTX 5080 16GB. Both give you comfortable 16 GB for QLoRA fine-tuning up to 13B, fast SDXL generation, and solid PyTorch training throughput. The 5070 Ti is the better price/performance pick; the 5080 gives you a bit more compute headroom.
Serious Local LLM and 30B Fine-tuning
RTX 5090 32GB. The 32 GB VRAM is the key advantage, not the compute. If you regularly work with Qwen 32B, Llama 70B Q4, or FLUX Dev FP16, the 5090 is the only consumer card that handles these without painful CPU offloading.
Enterprise and 70B+ Training
Cloud A100 or H100 via RunPod, Lambda Labs, or Vast.ai. No consumer card can handle full-precision 70B training. Rent cloud GPUs for training runs, use a local RTX 5090 for iteration and inference.
Frequently Asked Questions
Is the H100 worth buying outright for a home lab?
Almost certainly not. A used H100 SXM5 runs $25,000 or more and requires a server platform with NVLink. For the same money you could rent H100 cloud time for years. The H100 makes sense for companies running continuous multi-month training jobs, not for individual researchers.
Does the RTX 5090 compete with the A100?
For inference on models that fit in 32 GB, the 5090 comes surprisingly close to the A100. For training at scale, the A100’s HBM2e bandwidth (2,000 GB/s vs 1,792 GB/s) and ECC memory give it an edge. For anything requiring more than 32 GB VRAM, the A100 wins outright on capacity.
Should I buy two RTX 5080s instead of one RTX 5090?
Two RTX 5080s give you 32 GB total VRAM (16 GB each, not pooled unless using NVLink which the 5080 does not support), roughly double the compute, but VRAM is not shared transparently. For training with model parallelism, this can work. For inference, most tools require the model to fit on a single card. One RTX 5090 is simpler and gives you 32 GB on a single card with NVLink-compatible future expansion paths. See our CPU guide for platform requirements if you do go multi-GPU.
How do AMD GPUs perform for deep learning in 2026?
AMD’s ROCm stack has improved significantly. For inference with llama.cpp, AMD RX 9070 XT (16 GB) is competitive with NVIDIA equivalents at a lower price. For training with PyTorch, ROCm support is solid for standard workloads but custom CUDA kernels and some libraries still require NVIDIA hardware. For pure learning and standard training tasks, AMD is a viable option in 2026 if the price is right.
What about the RTX 5090 vs RTX 4090 for training?
The 5090 wins by 45-75% on training tasks thanks to 78% more memory bandwidth and 33% more VRAM. The 4090 remains competitive for 7B-13B workloads where 24 GB is sufficient, especially at used prices around $900. For a full comparison see our RTX 5090 vs 4090 deep dive.
Related Reading
Ready to Pick Your GPU?
Need help picking the right system? See our AI Workstation Build Guide for GPU, CPU, RAM, and storage recommendations together.