Quantization

PART 13 — Quantization

Model weights are usually saved as high-precision decimals (FP16: 16-bit floats). * Quantization: Rounding off those decimals to smaller numbers (e.g., INT8 or INT4).

Precision (How accurate the math is)
   ↕
Memory (Lower precision = less VRAM required)
   ↕
Speed (Smaller numbers = faster math & memory transfer)
   ↕
Quality (Rounding off numbers makes the model slightly dumber)

Why does an HLD engineer care?

A 70B parameter model in 16-bit requires ~140GB of VRAM (requires 2-4 expensive GPUs). If you quantize it to 4-bit, it requires ~35GB of VRAM (fits on 1 cheaper GPU). Quantization is a primary lever for reducing serving costs.

---