Quantization

PART 13 — Quantization

Quantization is aggressive data compression.

Example: A weight is stored as a 16-bit Float: 0.15234 We want to store it as an 8-bit Integer (INT8). 1. We find the max/min of the matrix (e.g., -1.0 to 1.0). 2. We map that range to -128 to 127. 3. 0.15234 becomes 19. 4. During math, we multiply by a scale factor to de-quantize it back to approximately 0.149.

HLD Impact: Memory is cut in half. The math is slightly less accurate, but inference speed doubles because moving 1 byte from VRAM to Compute Cores is twice as fast as moving 2 bytes.

---