Phase 10: Sandbox Destruction Labs & Failure Recovery Playbooks¶

Audience: Principal Site Reliability Engineer & Distributed Systems Architect
Environment: Isolated Sandbox (3-node cluster, throttled CPU / disk)

1. Destruction Lab 1: The Tombstone Avalanche (`TombstoneOverwhelmingException`)¶

1.1 The Objective¶

Demonstrate how frequent row deletions generate a dense accumulation of tombstones that abort reads and exhaust the JVM heap.

1.2 Execution Procedure¶

# Step 1: Create a queue-like table in CQL
cqlsh -e "
CREATE KEYSPACE lab WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};
CREATE TABLE lab.task_queue (
    tenant_id uuid,
    task_id bigint,
    payload text,
    PRIMARY KEY (tenant_id, task_id)
);
"

# Step 2: Insert 100,000 tasks into a single tenant partition
python3 -c '
from cassandra.cluster import Cluster
import uuid
cluster = Cluster(["127.0.0.1"])
session = cluster.connect("lab")
tenant = uuid.UUID("a82f0910-1234-5678-9abc-def012345678")
for i in range(100000):
    session.execute("INSERT INTO task_queue (tenant_id, task_id, payload) VALUES (%s, %s, %s)", 
                    (tenant, i, "task_payload"))
'

# Step 3: Delete 99,995 tasks, leaving only 5 live tasks
python3 -c '
from cassandra.cluster import Cluster
import uuid
cluster = Cluster(["127.0.0.1"])
session = cluster.connect("lab")
tenant = uuid.UUID("a82f0910-1234-5678-9abc-def012345678")
for i in range(99995):
    session.execute("DELETE FROM task_queue WHERE tenant_id = %s AND task_id = %s", (tenant, i))
'

# Step 4: Flush Memtable to create SSTable on disk
nodetool flush lab task_queue

# Step 5: Attempt to read the 5 remaining live tasks
cqlsh -e "SELECT * FROM lab.task_queue WHERE tenant_id = a82f0910-1234-5678-9abc-def012345678 LIMIT 5;"

1.3 Observed Failure Telemetry¶

Client throws instant fatal error:

ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read] 
message="Operation failed - received 0 responses and 1 failures: 
(ReadFailure due to tombstone overwhelming exception)"

Server log (/var/log/cassandra/system.log):

ERROR [ReadStage-2] 2026-10-15 09:45:10 SliceQueryFilter.java:195 - 
Scanned over 100001 tombstones in lab.task_queue; query aborted!

1.4 Production Incident Recovery Playbook¶

Do NOT use Cassandra as an active transactional task queue!
To recover access immediately, perform a Major Compaction with temporary zero grace seconds:
```
cqlsh -e "ALTER TABLE lab.task_queue WITH gc_grace_seconds = 0;"
nodetool compact lab task_queue
cqlsh -e "ALTER TABLE lab.task_queue WITH gc_grace_seconds = 864000;"
```
(Running compaction with gc_grace_seconds = 0 forces immediate purging of all tombstones from disk).

2. Destruction Lab 2: Compaction Lag & Read Latency Collapse¶

2.1 The Objective¶

Simulate a write-heavy workload with disabled compaction, generating hundreds of overlapping SSTables, and observe read latency collapse from \(2\text{ms}\) to over \(2,000\text{ms}\).

2.2 Execution Procedure¶

# Step 1: Intentionally disable autocompaction on target table
nodetool disableautocompaction lab task_queue

# Step 2: Ingest 50 batches of random writes, flushing to disk between each batch
for i in {1..50}; do
    python3 -c '
from cassandra.cluster import Cluster
import uuid, os
cluster = Cluster(["127.0.0.1"])
session = cluster.connect("lab")
for j in range(1000):
    session.execute("INSERT INTO task_queue (tenant_id, task_id, payload) VALUES (uuid(), %s, %s)",
                    (j, os.urandom(64).hex()))
    '
    nodetool flush lab task_queue
done

# Step 3: Count SSTables generated on disk
ls -l /var/lib/cassandra/data/lab/task_queue-*/nb-*-Data.db | wc -l
# (Displays 50+ individual SSTables!)

# Step 4: Measure read latency across multiple SSTables
cqlsh -e "TRACING ON; SELECT * FROM lab.task_queue LIMIT 10;"

2.3 Observed Failure Telemetry¶

Tracing output displays:

Executing single-partition query on lab.task_queue
Key cache miss...
Bloom filter evaluated on 50 SSTables: 48 false-negatives, 2 hits...
Seeking partition in nb-1-big-Index.db...
Seeking partition in nb-2-big-Index.db...
[Total Read Latency: 2,480 ms!]

nodetool tablehistograms lab task_queue:

SSTables per Read: Min: 45, P95: 50, Max: 50  <--- CATASTROPHIC!

2.4 Production Recovery Playbook¶

Re-enable autocompaction:

nodetool enableautocompaction lab task_queue

Accelerate compaction throughput to drain the backlog:
```
nodetool setcompactionthroughput 256
```
Monitor progress via nodetool compactionstats. Once compacted to 1 SSTable, read latency instantly drops back to \(< 2\text{ms}\)!