Phase 10: Sandbox Destruction Labs & Failure Recovery Playbooks¶

Audience: Principal Site Reliability Engineer & Distributed Systems Architect
Environment: Isolated Sandbox (3-node ZooKeeper Ensemble, throttled CPU / disk)

1. Destruction Lab 1: The Herd Effect Lock Stampede¶

1.1 The Objective¶

Demonstrate how a naive distributed lock design triggers a herd effect watch stampede that saturates the server CPU, then verify that Curator's sequential lock eliminates the stampede.

1.2 Execution Procedure¶

# Step 1: Simulate 500 parallel worker threads attempting a naive lock
# (All 500 threads create /naive_lock; 499 fail and register a watch on /naive_lock)
python3 -c '
from kazoo.client import KazooClient
import threading, time

def worker(i):
    zk = KazooClient(hosts="127.0.0.1:2181")
    zk.start()
    try:
        zk.create("/naive_lock", ephemeral=True)
        time.sleep(2) # Hold lock briefly
        zk.delete("/naive_lock")
    except Exception:
        # Failed to get lock; watch for deletion
        @zk.DataWatch("/naive_lock")
        def watch_lock(data, stat, event):
            pass
        time.sleep(5)
    zk.stop()

threads = [threading.Thread(target=worker, args=(i,)) for i in range(500)]
for t in threads: t.start()
for t in threads: t.join()
' &

# Step 2: In Terminal 2, monitor outbound packet spikes via mntr
watch -n 0.5 "echo mntr | nc 127.0.0.1 2181 | grep -E 'zk_packets_sent|zk_avg_latency'"

1.3 Observed Failure Telemetry¶

The moment Worker 1 deletes /naive_lock, zk_packets_sent surges by 500 packets in a single millisecond.
zk_avg_latency spikes from \(0.2\text{ ms}\) to \(> 140\text{ ms}\).
Server CPU spikes to 100% as the event thread iterates through the entire WatchManager table.

1.4 Production Mitigation Playbook¶

Replace naive locks with Curator's InterProcessMutex (fair lock using Ephemeral Sequential znodes). Each worker watches only its immediately preceding node, reducing network traffic from \(O(N)\) to strictly \(O(1)\)!

2. Destruction Lab 2: TxnLog Disk I/O Stall & Follower Quorum Eviction¶

2.1 The Objective¶

Simulate a slow disk or saturated EBS volume holding dataLogDir, observe fsync() taking longer than syncLimit, and verify that the Leader convicts and drops the Follower from the active quorum.

2.2 Execution Procedure¶

# Step 1: In Follower 2, freeze disk I/O on the transaction log directory
# (Using Linux cgroups or SIGSTOP on the sync thread)
sudo kill -STOP $(pgrep -f "QuorumPeerMain" | tail -n 1)

# Step 2: Bombard Leader with mutating write operations
python3 -c '
from kazoo.client import KazooClient
zk = KazooClient(hosts="127.0.0.1:2181")
zk.start()
for i in range(1000):
    zk.create(f"/bench/key_{i}", b"data", ephemeral=False)
'

# Step 3: Inspect Leader logs (/var/log/zookeeper/zookeeper.log)
tail -f /var/log/zookeeper/zookeeper.log

2.3 Observed Failure Telemetry¶

Leader attempts to send heartbeat PING packets to Follower 2.
Follower 2 does not respond within tickTime * syncLimit (10 seconds).

Leader log:

WARN  [LeaderHandler-/127.0.0.1:2888] Follower 2 failed to respond within syncLimit!
WARN  [LeaderHandler-/127.0.0.1:2888] Dropping Follower 2 from active forwarding list!

Run echo mntr | nc 127.0.0.1 2181:

zk_synced_followers   1   <--- DROPPED FROM 2 TO 1! (Quorum barely preserved!)

2.4 Incident Recovery Playbook¶

Resume process (sudo kill -CONT <pid>) or heal disk subsystem.
Follower reconnects to Leader.
Leader evaluates how many transactions Follower missed:
If missing \(< 100,000\) txns: Leader sends DIFF packet, streaming missing operations from RAM cache.
If missing \(> 100,000\) txns: Leader sends SNAP packet, transferring full snapshot to resync Follower cleanly!

3. Destruction Lab 3: Leader Sudden Death & Sub-500ms Quorum Failover¶

3.1 The Objective¶

Abruptly terminate the active ensemble Leader with SIGKILL, trace the Fast Leader Election protocol across remaining followers, and verify sub-500ms consensus restoration.

3.2 Execution Procedure¶

# Step 1: Identify the Leader PID
LEADER_PID=$(echo stat | nc 127.0.0.1 2181 | grep -q "Mode: leader" && echo "Node 1 is leader")

# Step 2: Kill Leader instantly with SIGKILL
sudo kill -9 $(pgrep -f "server.1")

# Step 3: Monitor Follower logs to measure exact failover duration
grep -E "LOOKING|LEADING|FOLLOWING|election" /var/log/zookeeper/zookeeper-server2.log

3.3 Recovery Telemetry & Handshake¶

09:50:00.100 WARN  [QuorumPeer[myid=2]] Lost connection to leader!
09:50:00.102 INFO  [QuorumPeer[myid=2]] State changed to LOOKING
09:50:00.105 INFO  [FastLeaderElection] Notification: Vote(leader=2, zxid=0x200000045, epoch=2)
09:50:00.210 INFO  [FastLeaderElection] Received majority votes (2/3) for Node 2!
09:50:00.320 INFO  [QuorumPeer[myid=2]] State changed to LEADING (Epoch 3 established!)
09:50:00.345 INFO  [Leader] Ready to process client transactions!

Total Failover Duration: 245 milliseconds!
Zero client session data loss; client drivers automatically reconnect to the newly elected leader within seconds!