Skip to content

Gpu Ai Infrastructure Basics

PART 12 — GPU / AI Infrastructure Basics

When a model is too big (e.g., 70B parameters = 140GB of VRAM), it doesn't fit on one 80GB GPU.

Tensor Parallelism Example: We have a massive weight matrix W sized 10000 x 10000. 1. We split W vertically. 2. GPU_0 gets columns 0-4999. 3. GPU_1 gets columns 5000-9999. 4. When input X arrives, it is broadcast to BOTH GPUs. 5. GPU_0 computes its half of the matrix math. GPU_1 computes its half. 6. Over a high-speed NVLink cable, they sum their results together in microseconds.