Skip to content

Gpu Ai Infrastructure Basics

PART 12 — GPU / AI Infrastructure Basics

  • CPU: A few very smart cores. Good for sequential logic (databases, web servers).
  • GPU: Thousands of dumb cores. Good for doing simple math on massive grids of numbers at the exact same time.
  • VRAM: GPU Memory. Where the model weights and KV cache live.
  • Memory Bandwidth: How fast data can move from VRAM to the GPU cores. (Decode phase is bound by this).
  • Compute: How fast the cores do math. (Prefill phase is bound by this).

Distributed AI: If a model (like Llama 3 70B) is too large to fit in one GPU's memory: * Tensor Parallelism (TP): Slicing the math matrices vertically. GPU 1 computes half the layer, GPU 2 computes the other half. They sync over lightning-fast cables. Low latency, but requires GPUs to be in the same physical box. * Pipeline Parallelism (PP): GPU 1 holds layers 1-40. GPU 2 holds layers 41-80. Data flows sequentially. Good for spanning across different physical machines.

Why does an HLD engineer care?

You cannot just autoscale AI instances like AWS EC2 web servers. Models have strict VRAM requirements. If you choose a 70B model, you must use Multi-GPU serving. Understanding TP vs PP helps you design node architectures.