Skip to content

Checksums

1. What is it? (2-3 lines)

A Checksum is a small, fixed-size hash value (e.g. CRC32, Adler-32, MD5) calculated from a block of digital data. It is transmitted or stored alongside the data to detect silent data corruption, bit rot, or network transmission errors.

2. Why does it exist?

Hardware storage media (HDDs, SSDs) and network infrastructure suffer from Silent Data Corruption (bit flips caused by cosmic rays, degraded flash cells, or noisy network hardware). Without Checksums, storage systems and networks read corrupted bytes as valid data, corrupting databases and system states without raising hardware exceptions.

3. Where is it used?

  • Distributed Storage & Filesystems: ZFS, Btrfs, HDFS, Apache Cassandra, AWS S3 (ETag / Content-MD5).
  • Network Protocols: TCP/UDP header checksums, TLS payload integrity checks.
  • Database Logs: PostgreSQL WAL block checksums, MySQL InnoDB page checksums.

4. How does it work? (simple flow)

  1. Write Time: Before writing a 4KB data block to disk or network, the engine passes the bytes through a non-cryptographic fast hash function: checksum = CRC32C(data_bytes).
  2. Storage: The 32-bit checksum integer is written to the page header or block metadata.
  3. Read Time: When reading the 4KB block back from disk/network, the engine recalculates current_checksum = CRC32C(read_bytes).
  4. Verification:
  5. current_checksum == stored_checksum \(\implies\) Data is valid.
  6. current_checksum != stored_checksum \(\implies\) Silent Data Corruption Detected! Engine throws a checksum error or repairs data automatically using replicas / Erasure Coding.

5. Advantages

  • Detects silent bit rot and network corruption before bad data infects database memory.
  • Extremely low CPU overhead when using hardware-accelerated CRC32C instructions (SSE4.2 / ARMv8).
  • Enables automated self-healing storage architectures (ZFS/HDFS fetch healthy replicas upon checksum failure).

6. Limitations

  • Non-Cryptographic Checksums (CRC32): Designed for error detection, NOT security! Malicious attackers can tamper with data and recalculate valid CRC32 checksums (requires Cryptographic Hashes / HMACs for security).
  • Adds minor storage overhead per page/block (e.g., 4-8 bytes per 4KB page).

7. Common interview questions (10)

  1. What is Silent Data Corruption (Bit Rot) and how do Checksums prevent it?
  2. Why is CRC32C preferred over MD5 or SHA-256 for storage checksums?
  3. How does ZFS use Merkle Trees of Checksums to provide self-healing storage?
  4. How do TCP and UDP checksums detect network packet corruption?
  5. What happens when PostgreSQL encounters a failed page checksum during a read operation?
  6. How does AWS S3 use MD5 / SHA256 checksums to verify multipart upload integrity?
  7. What is hardware-accelerated CRC32C instruction support in modern x86/ARM CPUs?
  8. How does Cassandra use block-level checksums to trigger Scrub and Read Repair operations?
  9. What is the difference between Error Detection (Checksums) and Error Correction (Reed-Solomon / ECC RAM)?
  10. How do Checksums detect torn page writes during database crashes?

8. Common misconceptions

  • Misconception: Checksums fix/repair corrupted data. Fact: Standard checksums only detect corruption. Repairing data requires Replicas, Erasure Coding, or ECC memory.
  • Misconception: CRC32 is a secure cryptographic hash. Fact: CRC32 is purely for hardware error detection; it is vulnerable to malicious tampering.

10. 30-second cheat sheet

  • Purpose: Detect Silent Data Corruption (Bit Rot) on disk & network.
  • Mechanism: Compute CRC32C(bytes) at write \(\to\) Store in header \(\to\) Re-verify at read.
  • Hardware Acceleration: CPU SSE4.2 / ARMv8 CRC32C instructions execute in sub-nanoseconds.
  • Repair Pair: Checksum = Detects bad data; Replication / Erasure Coding = Repairs bad data.