Phase 4: Storage Engine: TxnLog, Fuzzy Snapshots & WAL Internals

Audience: Principal Storage Systems Architect & Performance Tuning Specialist
Source References: zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/

---

1. The Write-Ahead Transaction Log (FileTxnLog)

Every state-mutating transaction accepted by ZooKeeper is appended to an on-disk write-ahead log (FileTxnLog) before being applied to the in-memory DataTree.

/var/lib/zookeeper/data/version-2/
├── log.100000001                 <--- Active Transaction Log (Preallocated 64MB)
├── log.100084021                 <--- Next Transaction Log Segment
├── snapshot.100000000            <--- Historical In-Memory Snapshot
└── snapshot.100084020            <--- Latest In-Memory Snapshot

1.1 The 64-Megabyte File Pre-Allocation Mechanism

Appending a single small record (e.g. 50 bytes) to a file on a standard Linux filesystem (ext4/xfs) causes two distinct disk operations: 1. Writing the data bytes to disk. 2. Modifying the filesystem inode metadata to record the file size expansion.

To eliminate filesystem inode metadata overhead and prevent file fragmentation on physical disks, ZooKeeper implements File Pre-Allocation:

// FileTxnLog.java pre-allocation logic
if (position + sz > currentFileSize) {
    currentFileSize = currentFileSize + preAllocSize; // Default: 64 MB
    fileChannel.write(ByteBuffer.allocate(1), currentFileSize - 1);
}

- ZooKeeper pre-allocates 64 Megabytes of zeros at the end of the log file. - Subsequent transactions overwrite the pre-allocated zeros sequentially. - The disk write head moves in a continuous sequential arc, achieving bare-metal NVMe write speeds!

---

2. Fuzzy Snapshots: Non-Blocking State Serialization

When transactions accumulate, replaying millions of log records from disk during server boot would take hours. ZooKeeper takes periodic point-in-time dumps of the in-memory DataTree called Fuzzy Snapshots (FileSnap.java).

┌────────────────────────────────────────────────────────────────────────┐
│ The Snapshot Trigger & Randomized Jitter                               │
├────────────────────────────────────────────────────────────────────────┤
│ A snapshot is initiated when transactions since last snapshot reach:   │
│ Threshold = (snapCount / 2) + Random(snapCount / 2)                    │
│ Default snapCount: 100,000 transactions                                │
└────────────────────────────────────────────────────────────────────────┘

Why the Random Jitter?
To prevent all 5 ensemble members from initiating intensive disk I/O snapshots at the exact same second, avoiding cluster-wide synchronized latency spikes!

2.1 Why Snapshots are "Fuzzy" (Inconsistent on Disk)

ZooKeeper does NOT acquire a global read lock on the DataTree during a snapshot: 1. A background thread iterates through the ConcurrentHashMap nodes and serializes them to snapshot.<zxid>. 2. Simultaneously, active client writes continue modifying znodes in memory. 3. The resulting snapshot file is point-in-time inconsistent ("fuzzy"). 4. How Consistency is Guaranteed:
The snapshot records the highest transaction ID processed when it started (snapshot.zxid). During server crash recovery, ZooKeeper loads the fuzzy snapshot, then replays all subsequent transactions from log.* whose \(\text{zxid} \ge \text{snapshot.zxid}\), bringing the tree to a mathematically deterministic state!

---

3. The Autopurge Daemon

Because ZooKeeper appends 64MB log segments continuously, an active cluster generates hundreds of gigabytes of logs per week.

# Production Maintenance Configuration (zoo.cfg)
autopurge.purgeInterval = 1     # Run purge daemon every 1 hour
autopurge.snapRetainCount = 5   # Keep only the 5 most recent snapshots and their logs

[!CAUTION] If autopurge.purgeInterval is not enabled, ZooKeeper will never delete old logs. Physical storage will fill to 100%, causing Linux to reject all write syscalls and abruptly crashing the ensemble!