DCGM GPU Telemetry in Practice
October 10, 2026
Goal
I wanted to stand up NVIDIA DCGM on a GPU host and watch real telemetry — not nvidia-smi snapshots, but the continuous counters (SM utilization, memory bandwidth, clocks, power, temperature) that production monitoring is built on. This is the tooling behind the metrics in my GPU Performance post.
What I did
- Installed DCGM 4.7.0 from the NVIDIA CUDA repository (Ubuntu, CUDA 13)
- Started
nv-hostengineand confirmed it discovered the GPU - Enabled persistence mode and DCGM process stats
- Ran health checks — overall status Healthy — and a level-1 diagnostic — Pass
- Watched live counters with
dcgmi dmon -e 100,150,155,203,204(SM clock, temperature, power, SM util, memory util) while running a sustained PyTorch workload: 8192×8192 FP32 matrix multiplication, 325 iterations
I put all the scripts and captured outputs in the ai-infrastructure-lab repo.
Measurements
| Metric | Idle | Under load |
|---|---|---|
SM util (DCGM_FI_DEV_SM_UTIL) | 0% | 93–100% |
Mem util (DCGM_FI_DEV_MEM_COPY_UTIL) | 0% | 29–57% |
| SM clock | 300 MHz | 540–930 MHz |
| Power draw | ~10.6 W | ~61–86 W (70 W TDP) |
| Temperature | 55 °C | 59–68 °C |
| Workload | — | 325 iters × 8192² FP32 matmul, 121 s |
| Achieved | — | ~2.95 TFLOPS (~36% of 8.1 FP32 peak) |
| GPU memory | — | 2.76 GiB steady, 0 OOMs |
Reading the pair the way my performance post describes: SM util pinned near 100% while memory util sat at 30–55% — textbook compute-bound. My SMs were saturated; memory bandwidth wasn’t the constraint. That’s the rule working on a real card, not just on paper.
What caught my eye was power: the T4’s 70 W envelope is the limiter here, with readings brushing 86 W under boost. On a bigger card with headroom, the same workload would clock higher and push more TFLOPS — the power wall I wrote about, showing up in my own data.
What broke
Nothing — clean run. DCGM installed from the repo without dependency issues, health checks passed on my first attempt, and the workload completed all 325 iterations with zero OOMs and zero GPU errors.
Next
- Point dcgm-exporter at Prometheus and build the Grafana dashboard — same counters, production pipeline
- Run the identical workload on a larger GPU and compare the SM/mem counter pair
- Add a bandwidth-bound pattern (large vector add) to watch the pair flip: low SM, high mem