GPU Performance: Why Utilization Lies

October 9, 2026

What it is

A GPU reporting 95% utilization is not necessarily delivering 95% of its potential. It might be running inefficient kernels, waiting on memory, or chewing through tiny batches with launch overhead between every one. The dashboard looks healthy. The workload misses its SLO anyway.

I think of it as two questions: how much work can the card do, and how much of that is your workload actually getting? Utilization answers the first — barely. This post is about the second.

Why it matters for AI infrastructure

GPU performance drives capacity planning, workload placement, and cost. And every GPU decision I’ve watched go wrong started with a misread metric. Someone saw high utilization and ordered more cards. Someone compared TFLOPS across generations and picked the wrong SKU. The numbers weren’t lying. They just weren’t measuring what anyone thought they were measuring.

Size on utilization alone and you’ll buy hardware to fix a software problem.

How it works — the metrics that matter

GPU utilization tells you how busy the card is, not how much useful work it delivers. nvidia-smi’s number is the fraction of the sample period during which at least one kernel was executing. That’s it.

I’ve seen cards at 95% utilization delivering a third of the throughput of cards at 60%. The difference was kernel efficiency — thousands of tiny launches versus a few well-fused kernels. One GPU was busy. The other was productive.

The two DCGM counters I actually trust:

DCGM_FI_DEV_SM_UTIL — were the streaming multiprocessors doing work?
DCGM_FI_DEV_MEM_COPY_UTIL — how much memory bandwidth was moving?

Read them as a pair. High SM, low memory: compute-bound, good shape. Low SM, high memory: bandwidth-bound — where most inference lives, streaming weights once per token. Both low: the problem is upstream. Data loading, CPU bottleneck, batching too small. Don’t tune the GPU; fix what’s feeding it.

The roofline model is worth five minutes. Performance is capped by the smaller of peak compute and (memory bandwidth × arithmetic intensity):

Performance ≤ min(peak compute, bandwidth × arithmetic intensity)

Training usually hits the compute ceiling; inference usually hits the bandwidth wall. Know which side you’re on before optimizing — the fixes are completely different.

Workload metrics beat device metrics. Tokens per second, time to first token, time per output token. A GPU can look healthy on every device counter while p99 latency blows the SLO. Device metrics describe the card. Workload metrics describe the service. Measure the service.

And the tradeoff that runs every inference deployment: batching. Bigger batches, more tokens per second, cheaper per token — but every request waits longer. No optimal batch size exists, only the one your SLO allows.

What the spec sheet hides

Clocks move. GPUs boost with headroom and throttle under power or thermal limits. Two identical cards in different cooling zones benchmark differently. Lock application clocks (nvidia-smi -lgc) before comparing hardware, or your benchmark measures the room.

Power limits are performance limits. A card pinned at its enforced power cap is leaving throughput on the table deliberately. Cooling and power provisioning are performance infrastructure, not facilities trivia.

Check the interconnect before blaming the GPU. NVLink versus PCIe is an order-of-magnitude gap for multi-GPU work. nvidia-smi topo -m shows what you actually bought.

The traps

What people say	What’s actually happening
“100% utilization, we’re maxed out”	Busy, not productive. Check tokens/sec or achieved FLOPS.
“High memory usage means a bandwidth bottleneck”	Capacity and bandwidth are different problems. Check which one you’re hitting.
“More GPUs mean proportionally more performance”	Communication and synchronization overhead eat the gains.
“2x the TFLOPS means 2x faster”	Different precisions, different sparsity — FP8 vs FP16 isn’t comparable.
“Our benchmark proves…”	With what clocks? Unlocked clocks make results unrepeatable.

Enterprise angle

GPU performance is a cost metric in disguise. Tokens per second per dollar, latency at a target load, throughput per watt — these tell you what the infrastructure is worth. Utilization alone doesn’t.

I’ve sat in rooms where someone defended a GPU fleet with utilization graphs. Technically accurate, completely meaningless. Before buying another GPU, ask the one question that changes capacity-planning decisions: is the hardware the bottleneck, or is the workload failing to use it effectively?