Phase 8: Hardware Sizing, JVM Tuning & Kernel Optimization¶

Audience: Principal Database Reliability Engineer & Infrastructure Architect
Source References: conf/zoo.cfg, bin/zkEnv.sh

1. The Golden Rule of Storage: Dedicated Disk for `dataLogDir`¶

In ZooKeeper, the leader cannot commit a transaction until a majority of followers acknowledge that the mutation is safely flushed to disk via fsync().

Disk I/O Contention Bottleneck:
[TxnLog Writes (Sequential, Low-Latency fsync)] ──┐
                                                 ├──► [SHARED HARD DRIVE / EBS VOLUME]
[Snapshots (Gigabyte Sequential Dump Every Hour)] ──┤
                                                 │
[OS System Logs & Monitoring Telemetry] ─────────┘

The Fatal Pathology:
When a periodic snapshot begins writing a 4GB dump to disk, the physical disk queue saturates. An incoming small transaction log write is blocked behind the snapshot. If the fsync() takes longer than tickTime * syncLimit (typically 10 seconds), the leader convicts the follower as dead and drops it from the quorum!

The Mandatory Production Architecture:¶

Separate disk paths in zoo.cfg:

# Fast, dedicated NVMe SSD mounted strictly for the append-only transaction log!
dataLogDir=/mnt/nvme-fast-disk/zookeeper/txnlog

# Standard storage mounted for fuzzy snapshots
dataDir=/var/lib/zookeeper/snapshots

2. JVM Heap Sizing & Garbage Collection Tuning¶

Unlike Elasticsearch or Cassandra (which require 32 GB heaps), ZooKeeper runs best on modest heaps (4 GB to 8 GB): - The entire DataTree is resident in RAM. - A 4 GB heap easily accommodates 3,000,000 znodes with ample headroom for session metadata. - Large heaps (32 GB+) increase the probability of long JVM Stop-The-World (STW) pauses. A 5-second GC pause causes the ensemble to lose quorum!

# JVM Flags (JVM.config or JAVA_OPTS)
-Xms4g
-Xmx4g
-XX:+UseG1GC
-XX:MaxGCPauseMillis=20
-XX:InitiatingHeapOccupancyPercent=45
-XX:G1ReservePercent=15
-XX:+ParallelRefProcEnabled

3. Host Linux Kernel & Swapping Mitigation¶

3.1 Eliminating Memory Swapping¶

# /etc/sysctl.conf
vm.swappiness = 1

If the Linux kernel swaps ZooKeeper heap pages to swap space on disk, response times jump from microseconds to milliseconds, causing peer election heartbeats to fail and triggering endless leader elections.

3.2 Network Socket Tuning (`zoo.cfg`)¶

tickTime=2000     # Fundamental unit of time (2 seconds)
initLimit=10      # Followers have 10 ticks (20s) to connect & sync to leader
syncLimit=5       # Followers have 5 ticks (10s) to reply to heartbeats
maxClientCnxns=2000 # Prevent rogue clients from exhausting file descriptors

4. Quorum Sizing & Multi-Datacenter Topology¶

Optimal Multi-Datacenter 5-Node Topology:
Datacenter 1 (Primary):    [ Node 1 (Voting) ]   [ Node 2 (Voting) ]
Datacenter 2 (Secondary):  [ Node 3 (Voting) ]   [ Node 4 (Voting) ]
Datacenter 3 (Arbiter):    [ Node 5 (Voting) ]

- If Datacenter 1 suffers a complete physical power outage, Datacenter 2 + Datacenter 3 hold 3 out of 5 votes, allowing the ensemble to elect a leader and continue operating without human intervention!