Phase 10: Sandbox Destruction Labs & Failure Recovery Playbooks

Audience: Principal Site Reliability Engineer & Streaming Architect
Environment: Isolated Sandbox (Standalone / MiniCluster, throttled CPU / memory)

---

1. Destruction Lab 1: Artificial Backpressure & Credit Pool Starvation

1.1 The Objective

Simulate a slow downstream sink, observe Credit-Based Flow Control propagate backpressure upstream, and verify credit pool exhaustion.

1.2 Execution Procedure

// Deploy a streaming job with an intentional artificial bottleneck
DataStream<Long> source = env.addSource(new RapidCounterSource()).setParallelism(2);

source.map(val -> val * 2).setParallelism(2)
      .addSink(new SinkFunction<Long>() {
          @Override
          public void invoke(Long value, Context context) throws Exception {
              // Simulate slow external API / database sink
              Thread.sleep(50); 
          }
      }).setParallelism(2);

1.3 Observed Failure Telemetry

• Open Flink Web UI -> Job Graph -> BackPressure:
• Sink: 0% Backpressure, 99% Busy Time.
• Map Operator: 98% Backpressure (RED).
• Source Operator: 98% Backpressure (RED).
• JMX / Prometheus Metrics:

  Map.numAvailableOutputBuffers = 0
  Map.backPressuredTimeMsPerSecond = 980
  Sink.numAvailableInputBuffers = 0 (Credits = 0!)
  

• Producer thread state transitions into TIMED_WAITING inside LocalBufferPool.requestMemorySegmentBlocking().

---

2. Destruction Lab 2: Checkpoint Alignment Collapse vs. Unaligned Rescue

2.1 The Objective

Under the backpressure generated in Lab 1, demonstrate that Aligned Checkpoints fail with alignment timeouts, then prove that Unaligned Checkpoints rescue the pipeline.

2.2 Execution Procedure

// Step 1: Run with ALIGNED checkpoints
env.enableCheckpointing(5000, CheckpointingMode.EXACTLY_ONCE);
env.getCheckpointConfig().setCheckpointTimeout(15000); // 15s timeout
// Unaligned checkpoints DISABLED by default

2.3 Failure Telemetry (Aligned)

• The checkpoint coordinator injects barriers into the source.
• Barriers are stuck behind thousands of buffered records in the network channel.
• Checkpoint UI shows:

  Checkpoint 101: FAILED
  Failure Reason: Checkpoint expired before completing.
  Alignment Duration: 15,002 ms (Timeout!)
  

• After 3 consecutive failures, the job crashes and restarts!

2.4 The Rescue (Unaligned Checkpoints)

// Step 2: Enable UNALIGNED checkpoints
env.getCheckpointConfig().enableUnalignedCheckpoints();

- Re-run under identical 50ms backpressure bombardment. - Observed Result: - The barrier jumps the network queues. - In-flight buffers are written directly into the checkpoint state file. - Checkpoint 101: COMPLETED in 340 milliseconds! Zero restarts, 100% resilient!

---

3. Destruction Lab 3: TaskManager Sudden Death & Zero-Loss Recovery

3.1 The Objective

Kill a running TaskManager process with SIGKILL, observe JobMaster failover detection, state reconstruction from durable storage, and verify zero data loss.

3.2 Execution Procedure

# Step 1: Submit a streaming pipeline with state tracking
# (Pipeline sums integers from Kafka into a RocksDB ValueState)

# Step 2: Identify TaskManager PID and terminate abruptly
ps aux | grep TaskManagerRunner
sudo kill -9 94812

# Step 3: Monitor JobManager logs
tail -f log/flink-*-standalonesession-*.log

3.3 Recovery Telemetry & Handshake

1. Heartbeat Timeout:

   WARN  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - 
   Lost connection to TaskExecutor 94812 (heartbeat timeout).
   

2. ExecutionGraph Failover:

   INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph - 
   Job 849f... switched to FAILING state.
   INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph - 
   Restarting from latest completed Checkpoint 241...
   

3. State Restoration:
4. JobMaster assigns tasks to a standby TaskManager slot.
5. RocksDB state backend downloads checkpoint SSTables from s3://flink-checkpoints/ (or local DFS).
6. Kafka Source seeks back to partition offsets committed in Checkpoint 241.
7. Stream processing resumes cleanly with Zero Data Loss and exact mathematical state integrity!