Gpu Ai Infrastructure Basics

PART 12 — GPU / AI Infrastructure Basics

Models are getting too big for one GPU (an H100 has 80GB VRAM; a 70B model needs ~140GB).

Distributed Inference: 1. Tensor Parallelism (TP): We slice the matrix math itself. GPU 1 computes W_left * x, GPU 2 computes W_right * x, and they sum the result. * SDE Analogy: Intra-request parallelism. Requires incredibly fast networking (NVLink) because they sync every millisecond. GPUs must be in the same physical server. 2. Pipeline Parallelism (PP): GPU 1 holds layers 1-40. GPU 2 holds layers 41-80. * SDE Analogy: Microservices. Request goes to Node A, then Node A passes it to Node B. Higher latency, but allows scaling across different physical servers via ethernet.

---