Skip to content

Flink vs Spark Streaming

1. Decision

Choose Apache Flink for ultra-low latency (<100ms), event-by-event true stream processing, complex event processing (CEP), and advanced stateful event-time windowing; choose Spark Structured Streaming for high-throughput micro-batch processing, unified batch and stream codebases, complex ETL transformations, and seamless integration with Spark MLlib and Delta Lake architectures.

2. 30-Second Answer

Apache Flink is a native, event-driven stream processing framework built on a Pipelined Execution Engine. It processes every incoming event continuously as an individual record in sub-millisecond time (\(<10\text{ms}\)). Flink provides first-class support for Event Time, complex out-of-order Watermarking, rich stateful aggregations backed by RocksDB, and Exactly-Once stateful processing via Chandy-Lamport lightweight snapshots. Spark Structured Streaming is built on top of Apache Spark's batch engine using a Micro-Batching Engine (discretizing continuous data streams into small sub-second batch intervals e.g. 100ms–1s). Spark delivers higher raw throughput per node, simpler unified Batch/Stream SQL APIs, and native integration with Delta Lake and ML pipelines, but incurs higher minimum latency (100ms+).

3. Decision Matrix

Option Best For Avoid When Latency Throughput State Management Operational Complexity
Apache Flink Ultra-low latency (<100ms), true real-time event processing, complex event processing (CEP), advanced session/sliding event-time windowing Simple nightly batch ETLs, workloads requiring unified batch/stream Spark MLlib pipelines Ultra-Low (<10ms event-driven streaming) High (Event-by-event pipelined engine) Advanced (Embedded RocksDB + Chandy-Lamport S3 snapshots) High (Flink cluster management, state savepoint migration tuning)
Spark Structured Streaming High-throughput streaming ETL, micro-batching (>100ms SLA), unified codebase for Batch & Streaming, Delta Lake integration Sub-millisecond strict real-time SLAs (e.g. real-time high-frequency trading or instant fraud detection) Low (100ms to 1s micro-batch interval) Extremely High (Amortizes execution overhead across micro-batches) Spark Checkpointing / HDFS state storage Medium

4. Selection Rules

Requirement Best Choice Rationale / Example
Real-Time Credit Card Fraud Detection (<50ms SLA) Apache Flink Flink's pipelined event engine evaluates fraud rules per event in sub-10ms
Streaming ETL to Delta Lake / Icehouse Spark Structured Streaming Micro-batching integrates natively with Spark DataFrame APIs and Delta ACID tables
Complex Event Processing (Detect Pattern A then B within 10s) Apache Flink (Flink CEP) Flink CEP library evaluates pattern matching across stateful event streams natively
Unified Batch and Stream Analytics Codebase Spark Structured Streaming Single PySpark/Scala API processes historical Parquet batch files and real-time Kafka streams

5. Tradeoffs

  1. Sub-Millisecond Latency (Flink) vs High Micro-Batch Throughput (Spark)
  2. Gain (Flink): True event-driven processing executes logic in \(<10\text{ms}\) per event.
  3. Lose (Flink): Slightly lower per-CPU throughput compared to micro-batching due to per-event processing overhead.
  4. Advanced Stateful Operations (Flink) vs Ecosystem Integration (Spark)
  5. Gain (Flink): RocksDB state backend handles terabytes of local key-value state seamlessly.
  6. Lose (Flink): Smaller ecosystem for machine learning compared to Spark MLlib and Databricks.
  7. Chandy-Lamport Snapshots vs Spark Checkpointing
  8. Gain (Flink): Continuous non-blocking state checkpoints with minimal latency impact.
  9. Lose (Spark): Micro-batch state checkpoints commit after every discrete batch interval.
  10. Developer Learning Curve
  11. Gain (Spark): Easy transition for existing Data Engineers familiar with PySpark/Spark SQL.
  12. Lose (Flink): Steeper learning curve mastering Flink DataStream APIs, ProcessFunctions, and State TTLs.
  13. Continuous Engine (Spark Continuous Processing Mode)
  14. Gain (Spark): Low-latency continuous processing mode in Spark reduces latency to ~1ms.
  15. Lose (Spark): Continuous mode in Spark has limited operator support and stability issues compared to native Flink.

6. Interview Counter Questions

  1. What is the fundamental architectural difference between Flink's Pipelined Engine and Spark's Micro-Batch Engine?
  2. How does Apache Flink achieve Exactly-Once processing semantics end-to-end?
  3. How does RocksDB serve as a state backend in Apache Flink and why is it critical for high-state applications?
  4. What is Spark Structured Streaming's Continuous Processing Mode and how does it differ from Micro-Batching?
  5. How do Flink Savepoints and Checkpoints differ during production deployments and cluster upgrades?
  6. How does Flink CEP (Complex Event Processing) pattern match sequences of events across time windows?
  7. What is the difference between Triggers and Watermarks in Spark Structured Streaming?
  8. How does Databricks Auto Loader simplify file streaming compared to Flink file sources?
  9. When would you choose Flink over Spark Streaming in an enterprise data platform?
  10. How do you monitor state size and Garbage Collection in Flink vs Spark?

7. Gold Standard Answers

  1. Flink: Continuous Pipelined Engine where threads pass records continuously through long-lived operator tasks (sub-10ms latency). Spark: Micro-Batch Engine where stream is discretized into tiny RDD batch jobs scheduled sequentially (100ms+ latency).
  2. End-to-end Exactly-Once requires: (1) Replay-able source (Kafka offsets), (2) Two-Phase Commit / Idempotent Sink (Kafka 2PC Sink / DB Upsert), (3) Flink ABS (Asynchronous Barrier Snapshotting) state checkpoints.
  3. RocksDB stores Flink state (e.g. window metrics) out-of-core on local NVMe SSDs instead of JVM Heap, preventing Stop-The-World GC pauses when state grows to terabytes.
  4. Continuous Processing Mode executes a long-lived epoch-based streaming engine in Spark achieving ~1ms latency, but supports limited SQL operators (no aggregations or join re-partitioning).
  5. Checkpoints: Automated, lightweight background state snapshots created for fault tolerance and recovery. Savepoints: Manually triggered, deterministic state snapshots used for cluster upgrades and schema migrations.
  6. Flink CEP defines pattern state machines (Pattern.begin("start").where(...).followedBy("end").within(Time.seconds(10))), matching complex sequences of multi-event conditions in memory.
  7. Watermarks: Track event-time progress to drop/handle late data. Triggers: Define when the output results of a micro-batch window should be evaluated and emitted to the sink.
  8. Databricks Auto Loader uses cloud notification queues (SQS/Event Grid) to incrementally ingest millions of files from S3/ADLS into Delta Lake efficiently without expensive file listing.
  9. Choose Flink when SLAs require \(<100\text{ms}\) execution latency, complex event-time session windowing, RocksDB stateful joins over multi-terabyte state, or complex event pattern matching (CEP).
  10. Flink: Monitor RocksDB memory usage, state size per key, and checkpoint duration. Spark: Monitor micro-batch execution time, memory spills to disk, and executor JVM GC pause metrics.

8. Common Mistakes

  • Choosing Spark Micro-Batching for high-frequency trading or instant fraud detection requiring sub-50ms latency.
  • Storing terabytes of state in Flink's Memory/FileSystem StateBackend instead of RocksDB, causing JVM Heap OOM crashes.
  • Upgrading Flink job code in production without creating a Savepoint, losing all active stateful stream window memory.
  • Using Spark Continuous Processing mode for complex multi-stream outer join SQL pipelines (unsupported operators).
  • Ignoring Kafka Sink 2PC configuration, causing duplicate message writes during Flink task restarts.

9. Keywords

Apache Flink, Spark Structured Streaming, Pipelined Engine, Micro-Batching, RocksDB, Chandy-Lamport, Asynchronous Barrier Snapshotting (ABS), Checkpoints, Savepoints, Flink CEP, Continuous Processing Mode, Delta Lake, Databricks Auto Loader, Two-Phase Commit Sink.

10. Cheat Sheet

  • Core Difference: Flink = Native event-by-event pipelined engine (\(<10\text{ms}\) true streaming); Spark Streaming = Micro-batch engine (\(100\text{ms}-1\text{s}\) discretized batches).
  • Latency Rule: \(<100\text{ms}\) SLA \(\rightarrow\) Apache Flink; \(>100\text{ms}\) SLA \(\rightarrow\) Spark Structured Streaming.
  • State Backend: Use RocksDB in Flink for out-of-core state storage on local NVMe SSDs to prevent JVM Heap GC crashes.
  • Upgrades: Always trigger a Savepoint in Flink before stopping a job to preserve state across code migrations.
  • Exactly-Once: Requires replay-able source (Kafka) + Flink ABS Checkpoints + 2PC/Idempotent Sink.
  • Unified Engine: Use Spark Structured Streaming if your team uses PySpark/Databricks and needs a single unified codebase for Batch & Streaming.

11. Related Decisions