Skip to content

Architectural Evolution of Apache Flink: 2010 to Present

Audience: Principal Distributed Systems Architect & Streaming Engine Specialist
Scope: 15-Year Evolution from Academic Research to Real-Time Cloud-Native Infrastructure

Chronological Evolutionary Timeline

2010: Stratosphere Project ──► TU Berlin research initiative on declarative data processing
2014: Open Sourced ──────────► Donated to Apache; incubated as Apache Flink (German for "agile")
2015: Flink 0.9 / 1.0 ───────► Asynchronous Barrier Snapshotting (ABS); DataSet vs DataStream API
2016: Flink 1.2 ─────────────► RocksDB State Backend; Rescalable State; Broadcast State
2017: Flink 1.4 ─────────────► Credit-Based Flow Control (Replaces legacy TCP-only backpressure)
2019: Flink 1.9 ─────────────► Blink engine merge (Alibaba contribution); Blink SQL planner
2020: Flink 1.11 / 1.12 ─────► Unaligned Checkpoints; Unified Batch/Streaming on DataStream API
2022: Flink 1.15 / 1.16 ─────► Changelog State Backend; Buffer Debloating; Adaptive Batch Scheduler
2024: Flink 2.0+ ────────────► Disaggregated State Storage; Cloud-Native Tiered Storage; Modern Java 17/21

1. Paradigm Shift 1: The Death of Lambda Architecture & The Rise of Kappa

1.1 The Lambda Architecture Curse (2011–2015)

In the early Big Data era, Nathan Marz popularized the Lambda Architecture to satisfy both speed and accuracy:

The Cumbersome Lambda Architecture:
                            ┌──► [Speed Layer: Storm / Samza] ──► Low Latency, Inaccurate State
                            │                                                 │
[Incoming Raw Events Stream]│                                                 ▼
                            │                                       [Serving Query Layer]
                            │                                                 ▲
                            └──► [Batch Layer: Hadoop / Spark] ──► High Latency, 100% Accurate

The Pain Points: 1. Two entirely separate codebases (Java/Scala for Storm + MapReduce/Spark for Batch) performing identical business logic. 2. Inevitable logic drift between batch and streaming pipelines. 3. Complex reconciliation logic in the serving layer to merge real-time deltas with batch historical snapshots.

Flink introduced the Kappa Architecture: - All data is inherently an infinite, continuous stream of events. - A "batch" dataset is simply a bounded stream with a known start and end. - A single stateful stream processing engine, if equipped with Event-Time processing and Exactly-Once fault tolerance, can serve both real-time fraud detection (\(10\text{ms}\) latency) and historical financial auditing with mathematical equivalence!

2. Paradigm Shift 2: API Unification (Deprecating the DataSet Duality)

In early versions (Flink 1.0–1.8), Flink maintained two distinct APIs: - DataSet API: Optimized for bounded batch datasets (using hash-joins and memory sorters). - DataStream API: Optimized for infinite real-time streams.

Flink unified the entire platform onto the DataStream API: 

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// Toggle between streaming and batch with a single runtime configuration!
env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC); // or BATCH / STREAMING
In BATCH mode: - Checkpoints are automatically disabled (fault tolerance relies on task replay). - State backends switch to fast, non-locking in-memory hash arrays. - Shuffle transitions from pipelined streaming to blocking persistent network partitions.

3. Paradigm Shift 3: The Checkpointing Revolution

Flink's original checkpointing engine relied on Aligned Barriers: - An operator with two input channels had to pause reading from Channel 1 the moment a barrier arrived until the matching barrier arrived on Channel 2. - The Catastrophic Failure: Under high backpressure, network buffers were completely clogged with data. Barriers took tens of minutes to travel down the network queues, causing checkpoints to time out and driving streaming pipelines into endless restart loops!

Aligned Checkpoint Barrier Stall:
Channel 1: [Barrier 100] ──► (Blocked! Must wait for Channel 2!)
Channel 2: [Data] [Data] [Data] [Data] [Data] ... [Barrier 100 is trapped behind 10,000 records!]
(Operator stalls; checkpoint alignment times out after 60 seconds!)

Unaligned Checkpointing fundamentally altered stream barrier semantics: - The moment a checkpoint barrier reaches the head of ANY input channel, it immediately jumps to the front of the queue. - The operator captures: 1. The operator's current internal state. 2. In-Flight Data Buffers: All unprocessed records in upstream input and output channels are serialized directly into the checkpoint file! - Result: Checkpoint duration becomes completely independent of network backpressure, completing in seconds even when pipelines are 100% saturated.

Historically, RocksDB checkpoints required taking a local snapshot and uploading SSTables. With the Changelog State Backend: - State mutations are continuously streamed to durable cloud storage (S3/HDFS) as an append-only changelog. - A checkpoint trigger merely writes a marker in the changelog, reducing checkpoint latency to milliseconds with virtually zero CPU or disk I/O overhead.

In 2019, Alibaba contributed its internal Blink engine back to Apache Flink: - Integrated Apache Calcite for advanced cost-based query optimization (CBO). - Introduced Dynamic Tables: $\(\text{Stream} \xrightarrow{\text{Continuous Query}} \text{Dynamic Table} \xrightarrow{\text{Changelog}} \text{Stream}\)$ - Provided native retract stream semantics (+I, -U, +U, -D) enabling SQL GROUP BY and multi-way streaming JOIN operations to emit correct mathematical aggregations over infinite data.

In Flink 1.x, RocksDB state is stored on local SSDs attached to TaskManagers: - Scaling up or rescheduling a TaskManager requires downloading gigabytes of state from S3 to local NVMe, causing minutes of rescheduling lag. - Fast-changing dynamic workloads struggle with static local storage allocations.

Flink 2.0 decouples compute from state storage: - State execution reads directly from a local shared cache while writing asynchronously to a disaggregated remote storage tier. - TaskManagers become near-stateless workers, enabling sub-second dynamic autoscaling in Kubernetes environments.