Quantization

PART 13 — Quantization

Floating point numbers take up space. float16 (FP16) takes 2 bytes per parameter.

• Quantization: Casting the data types down to int8 (1 byte) or int4 (0.5 bytes).
• Trade-off: You lose precision (the model gets slightly dumber), but you drastically reduce the VRAM required and increase memory bandwidth (smaller files load faster into compute cores).

SDE Takeaway: Quantization is a compression algorithm. It is the cheapest and most effective way to optimize inference infrastructure costs, allowing 70B models to run on consumer hardware.

---