Skip to content

Compression

1. What is it? (2-3 lines)

Compression is the process of encoding data using fewer bits than the original representation by identifying and removing statistical redundancies (lossless) or unperceptible information (lossy), reducing disk storage footprints and network transmission bandwidth.

2. Why does it exist?

Storage I/O and network bandwidth are major hardware bottlenecks in distributed systems. Compressing datasets (e.g. Parquet file blocks or Kafka message batches) reduces storage disk costs by 50-80% and drastically increases I/O throughput because reading smaller compressed bytes from disk/network is faster than CPU decompression overhead.

3. Where is it used?

  • Columnar Storage Formats: Apache Parquet, ORC, ClickHouse (Snappy, ZSTD, LZ4, Dictionary Encoding, Run-Length Encoding).
  • Message Brokers: Apache Kafka (compression.type=lz4 / zstd batch compression).
  • HTTP / Web Traffic: Gzip, Brotli (Content-Encoding: br).
  • Database Storage Engines: RocksDB / LevelDB (block-based LZ4/ZSTD compression).

4. How does it work? (simple flow)

  1. Generic Dictionary Encoding (LZ77 / Huffman): Identifies repeated byte strings, replacing duplicate byte sequences with short offset-length pointer tokens.
  2. Columnar Encoding Techniques:
  3. Dictionary Encoding: Replaces repetitive text columns (e.g. country_name) with tiny 1-byte integer keys.
  4. Run-Length Encoding (RLE): Replaces repeated identical values ('AAAABBB') with count-value tuples ((4,'A'), (3,'B')).
  5. Delta Encoding: Stores integer differences between consecutive values (1000, 1002, 1005 \(\to\) 1000, +2, +3).
  6. Decompression: Worker CPU reads compressed block from disk/network \(\to\) Expands byte tokens in RAM \(\to\) Returns original uncompressed stream.

5. Decision Matrix

Algorithm Speed (Compression) Speed (Decompression) Compression Ratio Primary Use Case
LZ4 Ultra-Fast Ultra-Fast (~4GB/s per core) Medium (~2x) Real-time RPCs, Kafka messages, RocksDB hot data
Snappy Fast Fast (~1GB/s per core) Medium (~2x–3x) Apache Spark, Parquet default, Bigtable
ZSTD (Zstandard) Tunable (Fast to Slow) Very Fast High (~3x–5x) Long-term log storage, Parquet/Iceberg cold files
Gzip / Deflate Medium Medium High (~3x–4x) HTTP Web assets, legacy log archiving

6. Limitations

  • CPU Overhead: Compression and decompression consume CPU clock cycles. If CPU is already pegged at 100%, adding compression degrades throughput.
  • Incompressible Data: Already-compressed binary files (JPEGs, MP4s, encrypted payloads) expand in size if compressed again.

7. Common interview questions (10)

  1. What is the tradeoff between CPU usage, compression ratio, and I/O throughput in distributed storage systems?
  2. Why is LZ4 or ZSTD preferred over Gzip in modern real-time streaming platforms like Apache Kafka?
  3. How does Columnar Storage (Parquet/ClickHouse) achieve dramatically higher compression ratios than Row-based databases?
  4. How do Dictionary Encoding, Run-Length Encoding (RLE), and Bit-Packing work in Apache Parquet?
  5. What is Delta Encoding and how does it compress timestamp/sequence column data?
  6. How does Kafka batch-level compression (compression.type=gzip/lz4/zstd) differ from individual message compression?
  7. What is the impact of HTTP Brotli (br) compression vs Gzip on web page load latency?
  8. Why should you NEVER compress already-encrypted or media files (JPEGs/MP3s)?
  9. How does ZSTD's dictionary-based compression optimize small JSON payload compression?
  10. How do RocksDB block-based compression algorithms operate across LSM Tree levels?

8. Common misconceptions

  • Misconception: Compression always slows down system performance. Fact: On modern multi-core CPUs, reading smaller compressed data from slow NVMe/S3 disk is far faster than reading raw uncompressed bytes, making compression a net performance WIN!
  • Misconception: Compress each row individually in a database. Fact: Individual row compression achieves terrible compression ratios; databases compress large 64KB–128MB block chunks or columnar batches.

10. 30-second cheat sheet

  • LZ4: Max Decompression Speed (~4GB/s). Best for Real-Time Streaming & Kafka.
  • ZSTD: Best Compression Ratio + High Speed. Modern standard for Data Lakes / Logs.
  • Snappy: Default for Google / Parquet / Hadoop.
  • Columnar Magic: Grouping identical data types together (Parquet) enables RLE + Dictionary encoding \(\implies\) 80%+ storage savings!