Apache Flink Mastery: From Stateful Stream Processing to Kernel Runtime Internals¶
10-Phase Distributed Streaming Architecture Curriculum¶
Target Audience: Principal Distributed Systems Architect, Stream Processing Engineer, Infrastructure Specialist
Depth: Cellular Anatomy, Asynchronous Chandy-Lamport Checkpoints, Credit-Based Flow Control, RocksDB Off-Heap State, Memory Pools, Unaligned Checkpoints
Curriculum Syllabus & Study Roadmap¶
| Phase | Module Title | File Link | Focus Areas |
|---|---|---|---|
| Phase 1 | Architecture, Dataflow & Runtime Model | phase_01_architecture_dataflow_runtime_model.md | Stratosphere origin (2010), JobManager, TaskManager, Task Slots, StreamGraph to ExecutionGraph compilation, Operator chaining |
| Phase 2 | Time Semantics, Watermarks & Windowing | phase_02_time_semantics_watermarks_windows.md | Event Time vs Processing Time vs Ingestion Time, Bounded-out-of-orderness watermarks, Late data handling, Window lifecycle |
| Phase 3 | Stateful Stream Processing & State Backends | phase_03_stateful_stream_processing_state_backends.md | Keyed State (ValueState, MapState), Operator State, HashMapStateBackend vs EmbeddedRocksDBStateBackend, State TTL |
| Phase 4 | Fault Tolerance & Chandy-Lamport Checkpoints | phase_04_fault_tolerance_chandy_lamport_checkpoints.md | Asynchronous barrier snapshotting (ABS), Aligned vs Unaligned checkpoints, Savepoints, Exactly-Once two-phase commit sinks |
| Phase 5 | Network Stack, Backpressure & Flow Control | phase_05_network_stack_backpressure_flow_control.md | Netty network stack, NetworkBufferPool, Credit-based flow control (Backlog + Credits), Detecting backpressure, Buffer debloating |
| Phase 6 | Distributed Execution, HA & Resource Management | phase_06_distributed_execution_ha_resource_management.md | Application vs Session vs Per-Job cluster mode, ZooKeeper/Kubernetes HA leader election, Slot sharing groups, Dynamic scaling |
| Phase 7 | Table API, Streaming SQL & Complex Event Processing | phase_07_table_api_sql_engine_cep.md | Dynamic Tables, Continuous Queries, Retract streams vs Append-only, Apache Calcite optimization, CEP pattern detection |
| Phase 8 | Production Memory Tuning (JVM + RocksDB) | phase_08_production_tuning_memory_jvm_rocksdb.md | Total Process Memory model (Framework, Task Heap, Managed Memory, Network, Off-heap), RocksDB native block cache tuning |
| Phase 9 | Real-World Choke Points & Diagnostics | phase_09_real_world_choke_points_diagnostics.md | Checkpoint alignment timeouts, Data skew in keyBy, Thread dump analysis, Flame graphs, Backpressure monitoring UI |
| Phase 10 | The Sandbox Destruction Labs | phase_10_destruction_labs_playbook.md | Lab 1: Artificial backpressure cascade, Lab 2: Checkpoint alignment timeout stall, Lab 3: RocksDB off-heap memory leak, Lab 4: TaskManager death & failover |
Companion Deep Dives & Revision Artifacts¶
- Master Connected Memory Map: Visual, graphical map connecting every Flink dot from stream sources and network buffers to RocksDB state and checkpoint storage.
- Architectural Evolution Deep Dive: 15-year engineering chronicle from Stratosphere research to unified streaming and Flink 2.0 disaggregated state.