Skip to content

Batch vs Stream Processing

1. Decision

Choose Batch Processing (Apache Spark, AWS EMR, Snowflake) for high-throughput, historical data analytics, complex multi-stage joins, and non-time-critical processing on bounded datasets; choose Stream Processing (Apache Flink, Spark Structured Streaming, Kafka Streams) for real-time, low-latency, event-by-event continuous processing on unbounded data streams (fraud detection, real-time alerting, IoT telemetry).

2. 30-Second Answer

Data processing architectures handle data in two fundamental modes based on data velocity and boundedness. Batch Processing operates on Bounded Datasets (fixed data files stored in S3/HDFS or SQL tables). Jobs run periodically (hourly, daily, nightly), reading massive datasets into memory to compute complex multi-stage aggregations, machine learning models, or monthly billing reports. It offers high throughput and complete dataset accuracy, but introduces processing latency delays (hours). Stream Processing operates on Unbounded Datasets (infinite, continuous streams of real-time events from Kafka/Kinesis). Events are processed individually or in sliding time windows within milliseconds of generation, enabling immediate fraud detection, real-time dashboards, and instant alerting, but requires complex state management and out-of-order event handling.

3. Decision Matrix

Option Best For Avoid When Processing Latency Scalability Data Bounds Operational Complexity
Batch Processing (Apache Spark / Snowflake) Nightly financial reporting, historical ML model training, complex multi-table joins, massive bulk data transformations Sub-second real-time fraud detection, instant live user dashboards, time-critical alerting High (Minutes to Hours; runs on scheduled batch triggers) High (Scales horizontally across massive compute clusters e.g. Spark worker nodes) Bounded Datasets (Fixed size files/tables) Medium
Stream Processing (Apache Flink / Kafka Streams) Real-time fraud detection, live user activity metrics, sliding window aggregations, IoT sensor processing Monthly billing reconciliations, complex deep historical data joins Ultra-Low (Sub-second milliseconds continuous execution) Extremely High (Scales stream partition consumption across worker nodes) Unbounded Datasets (Infinite continuous streams) High (State checkpointing, Event Time vs Processing Time, out-of-order watermarks)

4. Selection Rules

Requirement Best Choice Rationale / Example
Real-Time Credit Card Fraud Detection (<100ms SLA) Stream Processing (Flink) Evaluate sliding window transaction patterns continuously as events flow through Kafka
Nightly Financial Ledger Reconciliation & Reporting Batch Processing (Spark) Process 24 hours of closed transaction files stored in S3 at 02:00 AM
Live Sliding-Window Dashboard Metrics (Last 5 Mins) Stream Processing (Flink) Compute sum(revenue) over tumbling 5-minute event windows
Historical Machine Learning Model Training (5 Years Data) Batch Processing (Spark) Train Deep Learning model across 50TB of compressed Parquet files stored in Data Lake

5. Tradeoffs

  1. Sub-Second Real-Time Speed vs Maximum Throughput Efficiency
  2. Gain (Stream): Immediate insights within milliseconds of event generation.
  3. Lose (Stream): Lower total batch throughput per CPU cycle due to micro-batch/event-by-event overhead.
  4. State Management & Checkpointing Complexity
  5. Gain (Batch): Jobs are stateless between runs; failed batch jobs simply restart from scratch.
  6. Lose (Stream): Requires managing stateful RocksDB memory state and distributed Chandy-Lamport checkpoints.
  7. Bounded Complete Views vs Unbounded Out-of-Order Events
  8. Gain (Batch): Computes on complete, frozen datasets where late-arriving data is already static.
  9. Lose (Stream): Must handle late-arriving events, network delays, and out-of-order event timestamps using Watermarks.
  10. Compute Resource Costs
  11. Gain (Batch): Compute clusters spin up, execute the batch job, and immediately terminate (Serverless EMR).
  12. Lose (Stream): Stream processing workers must run continuously 24/7/365 in memory.
  13. Architectural Simplicity
  14. Gain (Batch): Simple SQL scripts and Spark DAG executions.
  15. Lose (Stream): Complex stream windowing semantics (Tumbling, Sliding, Session Windows).

6. Interview Counter Questions

  1. What is the difference between Bounded Datasets and Unbounded Datasets in Apache Beam / Flink?
  2. What is Event Time vs Processing Time vs Ingestion Time in Stream Processing?
  3. How do Watermarks handle late-arriving events in Apache Flink stream windows?
  4. What are the 3 main types of Windowing in Stream Processing (Tumbling, Sliding, Session)?
  5. How does Apache Flink implement Exactly-Once stateful stream processing guarantees (Chandy-Lamport algorithm)?
  6. What is Micro-Batching (Spark Streaming) vs True Event-Driven Streaming (Flink)?
  7. What is the Lambda Architecture vs Kappa Architecture?
  8. How do you handle stream state persistence (RocksDB state backend)?
  9. Why is replaying historical data harder in Stream Processing than in Batch Processing?
  10. How do you handle schema evolution in continuous streaming pipelines?

7. Gold Standard Answers

  1. Bounded: Fixed size data with clear start and end (e.g. static CSV file on disk). Unbounded: Infinite stream of continuous events with no end (e.g. live Kafka topic).
  2. Event Time: Timestamp when the event actually occurred on the client device. Ingestion Time: Timestamp when the message broker (Kafka) received the event. Processing Time: Timestamp when the stream worker executes the code.
  3. A Watermark is a temporal control signal embedded in event streams. A watermark \(W(t)\) asserts that the stream worker should assume no further events with EventTime < t will arrive, allowing the window to close.
  4. Tumbling: Non-overlapping fixed time windows (e.g. 00:00-00:05, 00:05-00:10). Sliding: Overlapping fixed time windows (e.g. 5-min window sliding every 1 min). Session: Gap-based activity windows separated by inactivity periods.
  5. Flink uses the Asynchronous Barrier Snapshotting (ABS) variant of the Chandy-Lamport algorithm: Checkpoint barriers flow alongside events down the stream. When a worker receives a barrier, it snapshots local RocksDB state to S3.
  6. Micro-Batching (Spark): Buffers incoming events into tiny 100ms-1s batch intervals (higher throughput, slightly higher latency). True Streaming (Flink): Processes each event individually as it arrives (sub-millisecond latency).
  7. Lambda: Dual-path architecture running both Batch layer (for accuracy) and Speed layer (for real-time). Kappa: Single-path architecture processing ALL data (real-time and historical replay) through a single Stream Engine.
  8. Flink stores active stream state (e.g. window accumulators) in an embedded RocksDB key-value store on local SSDs, asynchronously flushing state checkpoints to durable cloud storage (S3).
  9. Stream processing requires replaying historical Kafka event logs from offset zero into a new stream processing topic, which can consume massive CPU/RAM and temporary state storage.
  10. Use Apache Avro / Protobuf with Confluent Schema Registry. Streaming pipelines validate schema compatibility (Backward/Forward) before executing stream transformations.

8. Common Mistakes

  • Using Batch Processing for real-time fraud detection or instant alerting requiring sub-second SLAs.
  • Using continuous 24/7 Stream Processing for simple daily reports that could be computed via a 5-minute nightly batch job.
  • Relying on Processing Time instead of Event Time for stream windowing, skewing metrics during network delays.
  • Failing to configure Watermark lateness bounds in Flink, causing late-arriving network events to be silently dropped.
  • Storing stream state entirely in JVM Heap memory without RocksDB spillover, causing Stop-The-World GC crashes.

9. Keywords

Batch Processing, Stream Processing, Apache Spark, Apache Flink, Kafka Streams, Bounded Dataset, Unbounded Dataset, Event Time, Processing Time, Watermarks, Tumbling Window, Sliding Window, Session Window, RocksDB, Chandy-Lamport, Micro-Batching.

10. Cheat Sheet

  • Core Difference: Batch = Bounded historical data files processed periodically (High throughput, hours lag); Stream = Unbounded infinite events processed continuously (Sub-second latency).
  • Time Semantics: Always window on Event Time (device timestamp), NOT Processing Time! Use Watermarks to handle late-arriving events.
  • Window Types:
  • Tumbling: Fixed, non-overlapping (e.g. 00:00-00:05).
  • Sliding: Fixed, overlapping (e.g. 5-min window every 1 min).
  • Session: Dynamic gap-based activity windows.
  • State Storage: Store stream state in embedded RocksDB on local NVMe SSDs, periodically checkpointed to S3 via ABS/Chandy-Lamport.
  • Engine Standard: Flink = Sub-ms true streaming; Spark = High-throughput batch & micro-batching.

11. Related Decisions