Phase 10: Sandbox Destruction Labs & Failure Recovery Playbooks¶

Audience: Principal Site Reliability Engineer & Distributed Systems Architect
Environment: Isolated Sandbox (maxmemory 64mb, throttled CPU)

1. Destruction Lab 1: The Single-Thread Freeze & False Sentinel Failover¶

1.1 The Objective¶

Demonstrate how an unindexed $O(N)$ command freezes the single-threaded event loop, starves Sentinel health checks, and triggers an accidental, catastrophic cluster failover.

1.2 Execution Procedure¶

# Step 1: Populate Redis with 1,000,000 keys
redis-benchmark -p 6379 -r 1000000 -n 1000000 -d 50 -t set -q

# Step 2: In Terminal 1, simulate an application developer executing KEYS *
redis-cli -p 6379 KEYS * > /dev/null

# Step 3: Simultaneously in Terminal 2, measure client ping response
redis-cli -p 6379 --latency

1.3 Observed Failure Telemetry¶

Terminal 2 displays latency spiking from $0.2 \text{ ms}$ to $> 4,500 \text{ ms}$.
Sentinels fail to receive PONG responses within down-after-milliseconds 5000.

Sentinel logs show:

+sdown master mymaster 127.0.0.1 6379
+odown master mymaster 127.0.0.1 6379 #quorum 2/2
+try-failover master mymaster 127.0.0.1 6379

Sentinel elects a replica and issues SLAVEOF NO ONE. When the master finishes executing KEYS *, it is abruptly demoted and its dataset wiped by the incoming replica sync!

1.4 Production Mitigation Playbook¶

Disable KEYS entirely in configuration:
```
rename-command KEYS ""
```
Replace all batch scans in application code with the incremental, non-blocking cursor:
```
SCAN 0 MATCH user:* COUNT 100
```

2. Destruction Lab 2: The Fork Copy-on-Write (COW) Memory Explosion¶

2.1 The Objective¶

Trigger an Out-Of-Memory (OOM) kernel crash during RDB snapshotting by exploiting Linux Copy-on-Write memory amplification under heavy write mutations.

2.2 Execution Procedure¶

# Step 1: Enable Transparent Huge Pages on Linux
echo always | sudo tee /sys/kernel/mm/transparent_hugepage/enabled

# Step 2: Fill Redis to 80% of host RAM
# (In sandbox, configure maxmemory 40mb on a 64mb limited container)
redis-benchmark -p 6379 -n 200000 -r 100000 -t set -d 200 -q

# Step 3: Trigger a background save while continuously bombarding with random writes
redis-cli -p 6379 BGSAVE && \
redis-benchmark -p 6379 -n 500000 -r 100000 -t set -d 200 -q

2.3 Observed Failure Telemetry¶

As child process writes RDB, every write mutation triggers a 2 MB page copy.
Host physical RAM is instantly exhausted.

Kernel log (dmesg -T):

[kernel] Out of memory: Kill process 94812 (redis-server) score 950 or sacrifice child
[kernel] Killed process 94812 (redis-server) total-vm:84210kB, anon-rss:62940kB

Redis crashes instantly with zero logs, leaving corrupted temporary dump files.

2.4 Production Mitigation Playbook¶

Enforce strict THP disabling:

echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled

Tune Linux overcommit memory:
```
sudo sysctl -w vm.overcommit_memory=1
```
Set maxmemory to no more than 65% of physical host RAM to leave head-room for Copy-on-Write pages.

3. Destruction Lab 3: The Replication Buffer Storm Loop¶

3.1 The Objective¶

Induce an infinite replication failure loop where a replica repeatedly attempts full synchronization, crashes its output buffer on the master, disconnects, and restarts.

3.2 Execution Procedure¶

# Step 1: Configure an intentionally small replica output buffer on the master
redis-cli -p 6379 CONFIG SET client-output-buffer-limit "replica 10mb 5mb 10"

# Step 2: Bombard master with continuous write mutations
redis-benchmark -p 6379 -n 1000000 -t set -d 500 -P 16 -q &

# Step 3: Attach a new replica to initiate full sync
redis-server --port 6380 --replicaof 127.0.0.1 6379

3.3 Observed Failure Telemetry¶

Master begins generating RDB for replica.
Meanwhile, master write rate exceeds 10 MB in 5 seconds.
Mutations accumulate in the replica's output buffer.
Buffer crosses 10 MB hard limit.

Master log:

[server] Client id=142 addr=127.0.0.1:6380 scheduled to be closed:
         client-output-buffer-limit replica exceeded!
[server] Synchronization with replica 127.0.0.1:6380 failed!

Replica disconnects, reconnects immediately, triggers +FULLRESYNC again, and gets disconnected in an infinite CPU-thrashing loop!

3.4 Production Mitigation Playbook¶

Accurately size the replica output buffer to exceed the maximum write volume during the duration of an RDB dump: $$\text{Limit} > \text{Peak Write Rate (MB/s)} \times \text{RDB Generation Time (s)}$$

Configure:

client-output-buffer-limit replica 1gb 512mb 120
repl-backlog-size 512mb