Reading DCGM Counters

October 10, 2026

Reading DCGM Counters: How Two Numbers Tell Me What My GPU Is Doing

When I stood up DCGM in my lab, I had one question: how do I tell, from telemetry alone, whether a workload is starving the GPU or saturating it? The answer turned out to be two counters read as a pair — and my first measured run gave me a textbook example.

The two counters

DCGM exposes thousands of fields, but for performance diagnosis two carry most of the weight:

SM utilization (DCGM_FI_DEV_SM_UTIL) — the fraction of time at least one warp was active on a streaming multiprocessor. It tells me whether the compute units are busy.
Memory copy utilization (DCGM_FI_DEV_MEM_COPY_UTIL) — the fraction of time the memory system was moving data. It tells me whether the workload is feeding the SMs fast enough.

Either one alone lies. SM util at 100% sounds great — unless memory util is pinned at 100% too, in which case my SMs are busy waiting for data. So I read them as a pair: high SM + low mem means compute-bound (the math is the bottleneck); low SM + high mem means bandwidth-bound (the data path is the bottleneck); both low means the GPU is idle or the workload is broken somewhere upstream.

The worked example

My first measured run was an 8192×8192 FP32 matrix multiplication, 325 iterations on a Tesla T4, watched with dcgmi dmon -e 100,150,155,203,204 (SM clock, temperature, power, SM util, memory util). The counters settled into:

SM util: 93–100%, memory util: 29–57%
~2.95 TFLOPS sustained — about 36% of the T4’s 8.1 FP32 peak — over 121 seconds
SM clock floating 540–930 MHz, temperature 59–68 °C, zero GPU errors across all 325 iterations

Here’s what that looked like on my terminal — the moment the workload starts is the moment the idle zeros disappear:

dcgmi dmon -e 100,150,155,203,204
#Entity   SMCLK   TMPTR   POWER   GPUTL   MCUTL
ID                  C       W
GPU 0      300      55     10.65     0       0
GPU 0      300      55     10.55     0       0
GPU 0      600      59     68.92    93      45
GPU 0      555      60     73.04   100      44
GPU 0      585      60     68.24   100      57
GPU 0      615      61     68.04   100      46

(GPUTL is the SM utilization counter, MCUTL is memory copy utilization — dmon’s abbreviations aren’t obvious. Trimmed from the full capture; the complete run is in the lab repo as workload/dcgmi_dmon.out.)

SMs saturated, memory system half-idle: textbook compute-bound. Memory bandwidth wasn’t the constraint — the arithmetic was. That’s the pair rule from my GPU performance post working on a real card, not just on paper.

The surprise: the power wall

What caught my eye wasn’t in the utilization pair at all — it was power. The T4 is a 70 W card, and my readings brushed 61–86 W under boost. The card was riding its power envelope: no kernel optimization would have squeezed more TFLOPS out of that run, because there was no more power to clock higher with. This is the wall I wrote about in the performance piece — on modern GPUs the limiter is increasingly watts, not transistors. It also reframes “36% of peak”: the gap wasn’t my code, it was the envelope.

Full measurement table and reproduction scripts: DCGM GPU telemetry in the lab section.

What the pair doesn’t tell you

The pair rule got me the right diagnosis on my T4 run, but it has blind spots worth naming:

SM util isn’t occupancy. 100% SM util means at least one warp was active per SM — it says nothing about how many. A workload can sit at 100% with 10% occupancy, the SMs looking busy while mostly idling. For real efficiency, I measure achieved FLOPS against peak, not just the util counter.
One-second sampling hides bursts. dcgmi dmon samples every second by default. A kernel that pegs the GPU for 100 ms then idles for 900 ms reads as 10% — identical to a steady 10% workload, and a very different story. If the numbers look suspiciously smooth, I tighten the sample interval before trusting them.
Shared GPUs pollute the counters. On MIG partitions or time-sliced setups, SM and memory util are per-GPU, not per-workload — a neighbor workload’s numbers bleed into mine. Per-process stats or MIG-scoped fields are the fix.
Clocks need context. Low clocks don’t explain themselves. Thermal throttle, power cap, or just idle? I cross-reference with throttle violation counters and power draw before concluding anything about headroom.

None of these invalidated my run — but each one is a way the counters could have lied to me, and I didn’t know to check until I went looking.

What I’m running next

A bandwidth-bound pattern — large vector adds that barely touch the SMs — to watch the pair flip: low SM, high memory. If this post is the compute-bound case study, that run completes the pair. Same counters, same method, opposite diagnosis.