PyTorch Mixed Precision Training: FP16 vs BF16 Guide

PyTorch’s Automatic Mixed Precision (AMP) runs safe operations in 16-bit instead of 32-bit floating point, cutting memory use roughly in half and unlocking Tensor Core throughput on any GPU since Volta. Use torch.autocast with bfloat16 and skip GradScaler entirely if your GPU is Ampere or newer (RTX 30-series onward). Use float16 with GradScaler only if you are stuck on an older Turing or Volta card. NVIDIA’s own benchmarks show up to 3x training speedup from Tensor Core math alone, on top of the memory savings.

Quick Navigation:

What It Does FP16 vs BF16 Hardware Needed The Code

What Mixed Precision Actually Does

The key insight: “Mixed” precision means most of the network’s math runs in a 16-bit format while a few numerically sensitive operations stay in 32-bit. PyTorch’s autocast context manager decides which is which automatically, based on an internal list of operations that are safe to run at lower precision (matrix multiplies, convolutions) versus ones that need FP32’s range (softmax, loss computation, batch norm statistics).

Every floating point format trades range and precision for size. A standard FP32 number uses 1 sign bit, 8 exponent bits, and 23 mantissa bits. Cutting that to 16 bits does not just halve the size, it forces a choice about which 16 bits to keep.

Format	Bits	Exponent	Mantissa	Max Value
FP32 (standard)	32	8 bits	23 bits	~3.4 x 10^38
FP16 (half)	16	5 bits	10 bits	65,504
BF16 (bfloat16)	16	8 bits	7 bits	~3.4 x 10^38

FP16 keeps more mantissa bits (more precision per number) but only 5 exponent bits, so its range tops out around 65,504. Small gradient values common in deep learning underflow to zero well before that ceiling. BF16 keeps FP32’s full 8-bit exponent, giving it FP32’s dynamic range at the cost of 3 fewer mantissa bits. That single difference is why the two formats need completely different handling in practice.

Advertise on TensorRigs

This spot reaches ML engineers and AI hardware buyers. Get in touch

FP16 vs BF16

The short version

Use BF16 if your GPU supports it. It has FP32’s exponent range, so small gradients do not underflow to zero, which means you can skip GradScaler entirely. FP16 needs GradScaler because its narrow 5-bit exponent range causes small gradient values to flush to zero during backpropagation, silently losing updates. PyTorch’s own documentation is direct about this: GradScaler is “essential for float16 training but typically unnecessary for bfloat16.”

Why FP16 needs GradScaler

Gradients late in training are often very small. In FP16’s narrow range, values below about 6 x 10^-5 round to zero, and that update is lost. GradScaler multiplies the loss by a scale factor (commonly starting around 65,536) before the backward pass, pushing gradients into FP16’s representable range, then unscales them before the optimizer step. Skip this and training silently degrades or the loss goes to NaN.

Why BF16 does not need it

BF16’s exponent range matches FP32’s, so the same small gradients that underflow in FP16 remain representable in BF16. You lose some mantissa precision (7 bits instead of 10), which shows up as more rounding noise per number, but that noise rarely affects convergence the way a silently-zeroed gradient does.

When FP16 is still the right call

Older GPUs (Turing, Volta) have FP16 Tensor Cores but no native BF16 Tensor Core support, so BF16 math falls back to slower paths on those cards. If you are on one of those generations, FP16 with GradScaler is still the faster option, just watch for NaN losses.

Watch out for this specifically: PyTorch’s docs warn that “most bf16-pretrained models cannot operate in the fp16 numerical range… and will cause gradients to overflow instead of underflow.” If you are fine-tuning a model that was pretrained in BF16 (most large open LLMs are), switching it to FP16 for fine-tuning can cause the opposite problem, overflow instead of underflow. Match the fine-tuning precision to what the model was originally trained in when you can.

Hardware Requirements

Tensor Cores are the actual requirement

Autocast will run on any CUDA GPU, but the speedup depends entirely on Tensor Cores, specialized hardware units that accelerate 16-bit matrix math. Without them, FP16 and BF16 execute on the same CUDA cores as FP32, so you keep the memory savings but lose most of the speed gain. NVIDIA’s own mixed precision documentation puts Tensor Core throughput at up to 8x a standard FP32 pipeline.

Architecture	Example GPUs	FP16 Tensor Cores	BF16 Tensor Cores
Volta (2017)	V100	Yes	No
Turing (2018)	RTX 20-series, T4	Yes	No
Ampere (2020)	RTX 30-series, A100	Yes	Yes
Ada Lovelace (2022)	RTX 40-series	Yes	Yes
Blackwell (2025)	RTX 50-series	Yes	Yes

Every GPU on our GPU comparison page is Ampere or newer, so BF16 with no GradScaler is the right default for a new build. The only reason to reach for FP16 today is fine-tuning on older rented hardware, like a T4 or V100 instance from a budget cloud provider.

Advertise on TensorRigs

This spot reaches ML engineers and AI hardware buyers. Get in touch

Real Speedup Numbers

NVIDIA’s official mixed precision benchmarks report “up to 3x overall speedup on the most arithmetically intense model architectures,” with real per-model figures varying by how much of the workload is matrix-multiply-bound versus memory-bound:

Model	Framework	Speedup vs FP32
ResNet-50 v1.5	MXNet	3.47x
GNMT	PyTorch	2.35x
BERT Q&A	TensorFlow	1.94x

Why speedup varies: Models dominated by large matrix multiplies and convolutions (like ResNet’s convolutional layers) see the most benefit, since that is exactly what Tensor Cores accelerate. Models with more sequential, memory-bound operations see smaller gains because Tensor Core throughput was never the bottleneck for those layers. On top of the speed, expect activation memory to drop by roughly 40-50%, which is often the bigger practical win since it lets you raise batch size or fit a larger model.

The Code

BF16 on a modern GPU (Ampere or newer), no GradScaler needed:

import torch

model = model.to(“cuda”)

optimizer = torch.optim.AdamW(model.parameters())

for batch in dataloader:

optimizer.zero_grad()

with torch.autocast(device_type=“cuda”, dtype=torch.bfloat16):

outputs = model(batch)

loss = loss_fn(outputs, batch.labels)

loss.backward()

optimizer.step()

FP16 on an older GPU (Turing, Volta), GradScaler required:

import torch

model = model.to(“cuda”)

optimizer = torch.optim.AdamW(model.parameters())

scaler = torch.amp.GradScaler(“cuda”)

for batch in dataloader:

optimizer.zero_grad()

with torch.autocast(device_type=“cuda”, dtype=torch.float16):

outputs = model(batch)

loss = loss_fn(outputs, batch.labels)

scaler.scale(loss).backward()

scaler.step(optimizer)

scaler.update()

Combining with torch.compile: Wrap the compiled model call inside the autocast context, not the reverse. model = torch.compile(model) once outside the training loop, then call model(batch) as normal inside the with torch.autocast(...) block shown above. The two features are independent and PyTorch recommends using both together for training speed.

Common Pitfalls

Forgetting to unscale before gradient clipping

If you clip gradient norms with FP16 and GradScaler, call scaler.unscale_(optimizer) before torch.nn.utils.clip_grad_norm_. Clipping scaled gradients clips against the wrong threshold, since the values are inflated by the scale factor.

Mixing precision on the model’s parameters

Autocast only affects operations inside its context, not the stored parameters. Keep model weights and the optimizer’s state in FP32 (the default). Autocast casts activations on the fly during the forward pass, it does not need or want you to manually cast the model itself to FP16 or BF16.

Expecting inference speedup for free

Autocast works for inference too (wrap the forward pass, skip GradScaler entirely since there is no backward pass), but for serious inference workloads, dedicated quantization to INT8 or INT4 usually beats FP16/BF16 on both speed and memory. See our quantization guide for that comparison.

Frequently Asked Questions

Does mixed precision hurt model accuracy?

With AMP done correctly, the accuracy difference is usually negligible. Autocast keeps numerically sensitive operations like softmax and batch norm reductions in FP32 automatically, only casting the safe operations (mostly matrix multiplies and convolutions) to FP16 or BF16. Real accuracy loss almost always comes from skipping GradScaler with FP16, not from the mixed precision approach itself.

Do I need GradScaler with bfloat16?

No. GradScaler exists to prevent FP16 gradients from underflowing to zero, because FP16 only has 5 exponent bits and a maximum value of about 65504. BF16 uses the same 8 exponent bits as FP32, so it has FP32’s dynamic range and does not underflow the same way. Use torch.autocast with bfloat16 and skip GradScaler entirely.

My GPU doesn’t have Tensor Cores. Is AMP still worth using?

Barely. Without Tensor Cores, FP16 and BF16 math still executes, but on the same CUDA cores as FP32, so you lose the throughput gain and keep only the memory savings. Every GPU since Volta (2017) has Tensor Cores, so this mostly affects very old hardware. Everything currently sold on this site (RTX 40/50 series and newer workstation cards) has them.

Can I just use both FP16 and BF16 at once?

No, autocast uses one dtype per context. Pick BF16 if your GPU supports it (Ampere or newer) since it needs no GradScaler and is more numerically forgiving. Fall back to FP16 with GradScaler only on Turing or Volta cards that lack BF16 Tensor Core support.

Does torch.compile work with autocast?

Yes, and PyTorch recommends combining them. Wrap the compiled model call inside the autocast context, not the other way around. The two optimizations are independent: compile fuses and optimizes the computation graph, autocast controls per-operation precision. Using both together is the standard recommended setup for training speed in current PyTorch versions.

Need a GPU That Actually Supports BF16?

Compare GPUs

Browse Complete Builds

Every Ampere-or-newer card on our GPU comparison page gets full BF16 Tensor Core acceleration.