Phase 10: Sandbox Destruction Labs & Failure Recovery Playbooks¶

Audience: Principal Site Reliability Engineer & Distributed Systems Architect
Environment: Isolated Sandbox (cacheSizeGB: 1, throttled CPU / disk IOPS)

1. Destruction Lab 1: Dirty Cache Saturation & Application Thread Hijacking¶

1.1 The Objective¶

Demonstrate how an unthrottled write surge under constrained disk write throughput causes WiredTiger dirty cache to cross the 20% critical threshold, forcing client application threads to perform synchronous disk eviction and spiking latency by 5,000x.

1.2 Execution Procedure¶

# Step 1: Configure an intentionally small WiredTiger cache (512MB) in mongod.conf
# storage.wiredTiger.engineConfig.cacheSizeGB: 0.5

# Step 2: In Terminal 1, initiate a massive parallel unindexed write bombardment
mongoperf <<EOF
{
  "nThreads": 16,
  "fileSizeMB": 2048,
  "sleepMicros": 0
}
EOF

# Step 3: Concurrently bombard MongoDB with document inserts via python/node
python3 -c '
import pymongo, os
client = pymongo.MongoClient("mongodb://127.0.0.1:27017")
coll = client.test_db.heavy_writes
for i in range(1000000):
    coll.insert_one({"payload": os.urandom(2048), "idx": i})
' &

# Step 4: In Terminal 2, monitor mongostat
mongostat 1

1.3 Observed Failure Telemetry¶

mongostat output:

insert query update delete getmore command dirty  used flt vsize  res qrw  arw
  4500     0      0      0       0     2|0  18.5 78.2   0 1.2G 640M 0|0  0|0
  1200     0      0      0       0     2|0  22.8 89.4   0 1.2G 640M 5|28 12|14 <--- DIRTY > 20%!
    45     0      0      0       0     2|0  24.1 94.2   0 1.2G 640M 82|140 45|80 <--- STALL!

Application insert latency jumps from \(0.8\text{ ms}\) to \(> 4,800\text{ ms}\).
The arw (active read/write) and qrw (queued read/write) numbers explode as application threads stall inside __wt_page_evict().

1.4 Production Mitigation Playbook¶

Tune WiredTiger eviction server aggression in mongod.conf:

setParameter:
  wiredTigerConcurrentWriteTransactions: 128
  wiredTigerEngineRuntimeConfig: "eviction_dirty_target=5,eviction_dirty_trigger=15"

Provision higher IOPS storage (e.g. migrate from AWS gp2 to io2 Block Express).
Scale horizontally by sharding the collection across multiple replica sets.

2. Destruction Lab 2: Oplog Depletion & Stale Secondary Failure¶

2.1 The Objective¶

Sever the network connection of a secondary, exhaust the primary's Oplog window with rapid mutations, restore the connection, and observe the secondary enter the unrecoverable RECOVERING state.

2.2 Execution Procedure¶

# Step 1: Check current Oplog size and window
rs.printReplicationInfo();

# Step 2: Sever network to Secondary node (port 27018) using iptables
sudo iptables -A INPUT -p tcp --dport 27018 -j DROP

# Step 3: Bombard Primary with write operations exceeding the configured Oplog size
python3 -c '
import pymongo, os
client = pymongo.MongoClient("mongodb://127.0.0.1:27017")
coll = client.test_db.oplog_filler
for i in range(500000):
    coll.insert_one({"data": os.urandom(4096)})
'

# Step 4: Verify Primary Oplog has wrapped around (oldest entry overwritten)
rs.printReplicationInfo();

# Step 5: Heal the network partition
sudo iptables -D INPUT -p tcp --dport 27018 -j DROP

2.3 Observed Failure Telemetry¶

Inspect secondary status: rs.status():

{
  "name": "127.0.0.1:27018",
  "stateStr": "RECOVERING",
  "errmsg": "remote oplog does not contain entries as recent as our most recent entry. Oplog window exceeded. Cannot catch up.",
  "optime": { "ts": Timestamp(1716380100, 1), "t": 1 }
}

The secondary remains stuck in RECOVERING indefinitely and cannot serve reads or vote in elections.

2.4 Production Incident Recovery Playbook¶

Trigger Initial Sync on the Stale Secondary:

# Connect to the stale secondary node
# Stop mongod
sudo systemctl stop mongod

# Wipe the corrupted/stale data directory completely
rm -rf /var/lib/mongodb/*

# Restart mongod (Node will automatically initiate full Initial Sync from Primary)
sudo systemctl start mongod

Prevent Future Depletion: Expand the Oplog size to guarantee at least 48 to 72 hours of retention during maintenance windows:
```
db.adminCommand({ replSetResizeOplog: 1, size: 204800 }); // 200 GB
```

3. Destruction Lab 3: Split-Brain Network Partition & Rollback Forensic Extraction¶

3.1 The Objective¶

Simulate a network partition that isolates the primary, accept uncommitted non-majority writes, heal the partition, and extract the rolled-back data from the physical disk files.

3.2 Execution Procedure¶

# Step 1: Isolate Primary (Node A) from Secondaries (Nodes B and C)
sudo iptables -A OUTPUT -d <Node_B_IP> -j DROP
sudo iptables -A OUTPUT -d <Node_C_IP> -j DROP

# Step 2: Client writes a critical transaction to Node A with { w: 1 }
# (Write succeeds on Node A but never reaches Node B or C)
mongo --host 127.0.0.1:27017 <<EOF
db.orders.insertOne({ _id: 9999, customer: "VIP Client", status: "LOST_DATA" });
EOF

# Step 3: Nodes B and C form a quorum and elect Node B as new Primary
# New writes arrive on Node B:
mongo --host 127.0.0.1:27018 <<EOF
db.orders.insertOne({ _id: 10000, customer: "New Primary Client", status: "VALID" });
EOF

# Step 4: Heal network partition
sudo iptables -D OUTPUT -d <Node_B_IP> -j DROP
sudo iptables -D OUTPUT -d <Node_C_IP> -j DROP

3.3 Forensic Extraction & Data Recovery Playbook¶

Node A reconnects, steps down, detects the divergence, and automatically writes the lost document to disk:

# 1. Locate the generated rollback BSON file in the data directory
cd /var/lib/mongodb/rollback/
ls -lh ecommerce.orders/

# 2. Inspect the rolled-back document contents using bsondump
bsondump ecommerce.orders/removed.2026-10-15T09-35-12.0.bson

# Output shows:
# {"_id":9999,"customer":"VIP Client","status":"LOST_DATA"}

# 3. Manually re-inject the recovered document into the active Primary
mongorestore --host 127.0.0.1:27018 \
             --db ecommerce \
             --collection orders \
             ecommerce.orders/removed.2026-10-15T09-35-12.0.bson

Incident Resolved: 100% data fidelity restored with complete audit verification!