GPU Architecture

September 27, 2026

What it is

Think of the H100 as a factory with thousands of small workers. It works because the workers are organized in layers and kept supplied with data.

NVIDIA H100 SXM5 conceptual architecture

A GPU does one thing well: run the same operation on huge amounts of data at once. Matrix multiplication — which is most of what AI workloads actually are — is a perfect fit. Everything in this diagram exists to keep thousands of math units busy and fed.

One note before we go in: this is a conceptual view, not the physical silicon layout. Exact counts vary by SKU — the PCIe H100 ships with fewer SMs enabled than the SXM5. The ideas don’t change; the numbers shift.

From the outside in

Host and GPU links. PCIe Gen5 connects the GPU to the CPU, which hands it work and data. NVLink 4 connects GPUs directly to each other, so multi-GPU training doesn’t bottleneck on the CPU or PCIe. Two very different pipes — 128 GB/s versus 900 GB/s bidirectional. That gap shows up in every multi-GPU benchmark that matters.

GPCs — the unit of scale. The GH100 die is divided into 8 Graphics Processing Clusters. Each GPC holds 9 TPCs, each TPC holds 2 SMs — 8 × 9 × 2 gets you to 144. A GPC is mostly an organizational unit: shared front-end hardware, and a boundary NVIDIA draws the line at when binning. The shipping H100 SXM5 has 132 of the 144 SMs enabled. The disabled ones are typically defective units — that’s not a flaw in the story, it’s how you get usable yield out of an 814 mm² die. Same silicon, different SKUs, with the lines drawn at GPC and SM boundaries.

For compute workloads, honestly, you can think of the GPC as packaging. The SM is where the action is.

SMs — where the real work happens. Each streaming multiprocessor is a self-contained mini-processor:

128 CUDA cores for general floating-point math
4 Tensor Cores that multiply small matrix blocks in a single operation — the main source of AI speed. Everything else on this die is logistics; the Tensor Cores are the factory floor.
4 warp schedulers. Threads run in groups of 32 called warps. When one warp stalls waiting on memory, the scheduler swaps in another that has its data ready. The GPU doesn’t make memory faster — it just never sits idle waiting for it.
256 KB register file holding each thread’s private values — the fastest storage on the chip, and the smallest.
256 KB of L1 cache / shared memory. Shared memory is the fast scratchpad that threads in the same block use to cooperate. Getting data into shared memory and reusing it there is half of GPU programming.

L2 cache — 50 MB, shared by every SM. Miss in an SM’s L1 and the request lands here. A hit saves the slow trip out to HBM. 50 MB sounds generous until 144 SMs are all asking at once.

HBM3 — the memory wall, solved with physics. Five stacks, 16 GB each, 80 GB total, moving ~3.35 TB/s. What makes HBM different from the GDDR in a gaming card: the DRAM dies are stacked vertically with through-silicon vias and placed on a silicon interposer right next to the GPU die. Short distance plus a 5120-bit-wide interface (1024 bits per stack × 5) gives roughly 3x the bandwidth GDDR6X can manage.

The tradeoff is real: HBM costs more per gigabyte and you get less of it. NVIDIA chose bandwidth over capacity because AI workloads are bandwidth-starved. A 70B-parameter model in FP16 needs ~140 GB just for weights — more than fits — so it gets sharded, quantized, or offloaded, and what remains streams through that 3.35 TB/s pipe once per token. That streaming is the physical reason inference is bandwidth-bound. The H200 pushes further with HBM3e at 141 GB, but the architecture story is identical.

Why the layers exist

Every layer trades size for speed:

Level	Size	Relative speed
Registers	256 KB per SM	Fastest
L1 / shared memory	256 KB per SM	Very fast
L2 cache	50 MB, whole chip	Fast
HBM3	80 GB	Slowest (still very fast)

Fast GPU code keeps data as high in this table as possible and reuses it before fetching more. That’s not an optimization tip — it’s the whole game.

A quick example

Matrix multiply. The work splits into tiles, one per SM. Each SM pulls its tiles from HBM through L2 into shared memory, where its threads reuse them many times. Tensor Cores do the multiplying while warp schedulers hide the memory stalls. Results flow back out the same path. Every layer in the table earns its keep in those few milliseconds.

Reading my own metrics off this diagram

The two DCGM counters from the performance post live here: SM_UTIL measures the SM blocks — are the workers busy? MEM_COPY_UTIL measures the HBM at the bottom — is data moving? Read them as a pair and this diagram tells you why.

Enterprise angle

When someone quotes GPU specs, map each number onto this diagram. More SMs? More HBM? More bandwidth? Bigger L2? “2x faster” without a location on this diagram is marketing. With one, it’s architecture — and you can check whether the claim matches your workload’s actual bottleneck.