Skip to content

Lambda vs Kappa Architecture

1. Decision

Choose Kappa Architecture as the modern standard for real-time and historical analytics pipelines, processing ALL data through a single stream processing engine (Apache Flink / Spark) and an append-only event log (Apache Kafka); choose Lambda Architecture only for legacy environments where low-latency streaming results must be reconciled against an authoritative, complex batch processing layer.

2. 30-Second Answer

Lambda Architecture (pioneered by Nathan Marz) runs two separate parallel processing paths: (1) Batch Layer (Apache Hadoop/Spark) which periodically processes raw data to build accurate, batch views, and (2) Speed Layer (Apache Storm/Flink) which processes real-time streams to provide low-latency speed views. A Serving Layer merges batch and speed views at query time. While robust, Lambda creates code duplication (writing logic twice in Java and Spark) and eventual reconciliation bugs. Kappa Architecture (pioneered by Jay Kreps) eliminates the dual-path complexity: ALL data (both real-time streams and historical backfills) is processed through a Single Streaming Pipeline (Apache Flink) backed by an immutable, long-retention event log (Apache Kafka).

3. Decision Matrix

Option Best For Avoid When Latency Code Maintainability Data Reconciliation Operational Complexity
Kappa Architecture (Single Stream Path) Modern real-time data platforms, unified stream/historical processing (Flink + Kafka), log-centric event architectures Environments with short Kafka log retention where replaying 5 years of historical data is impossible Ultra-Low (<10ms real-time; fast stream replay for backfills) Excellent (Single codebase for real-time and historical data processing) High (Single source of truth in stream processing logic) Low to Medium
Lambda Architecture (Dual Batch + Speed Path) Legacy big data systems where batch processing requires complex multi-hour graph algorithms Modern real-time data platforms (Creates nightmare dual-codebase maintenance and data drift) Low for Speed layer; High for Batch layer Poor (Duplicated processing logic across Batch and Speed codebases) Complex (Reconciling Batch views vs Speed views causes data drift bugs) High

4. Selection Rules

Requirement Best Choice Rationale / Example
Modern Real-Time Data Pipeline Architecture Kappa Architecture Process live Kafka events via Flink; replay historical Kafka topics for backfills
Single Codebase for Real-Time & Historical Data Kappa Architecture Flink pipeline processes live events and historical backfills using identical code
Legacy Batch System Reconciling Real-Time Stream Lambda Architecture Batch Spark job runs nightly at 02:00 AM to overwrite approximate real-time Storm views
Historical Data Replay for Pipeline Code Upgrades Kappa Architecture Spin up a new Flink job instance; re-process raw Kafka log from offset zero

5. Tradeoffs

  1. Single Unified Codebase (Kappa) vs Dual Layer Resilience (Lambda)
  2. Gain (Kappa): Write business logic ONCE in Flink/Spark; zero code duplication.
  3. Lose (Kappa): High reliance on stream engine stateful processing correctness.
  4. Elimination of Data Drift vs Complex Log Retention
  5. Gain (Kappa): Prevents data discrepancies caused by Batch and Speed layers computing different answers.
  6. Lose (Kappa): Requires long-retention event logs (Kafka / Apache Pulsar / S3 tiering).
  7. Operational Simplicity vs Storage Costs
  8. Gain (Kappa): Operate 1 stream processing cluster instead of separate Batch + Speed + Serving systems.
  9. Lose (Kappa): Retaining petabytes of historical logs in Kafka clusters is expensive without Tiered Storage.
  10. Historical Backfilling Speed
  11. Gain (Lambda): Optimized batch engines (Spark / Snowflake) process 5 years of static files in parallel fast.
  12. Lose (Kappa): Replaying multi-year event streams through Flink requires high stream parallelism.
  13. Schema Migration Flexibility
  14. Gain (Kappa): To add a feature, launch a parallel stream job, replay from offset zero, and swap serving views.
  15. Lose (Kappa): Requires careful state savepoint management during job re-plays.

6. Interview Counter Questions

  1. What is the fundamental difference in data flow between Lambda Architecture and Kappa Architecture?
  2. Why is Code Duplication and Data Drift the biggest flaw of Lambda Architecture?
  3. How does Kappa Architecture perform historical data backfills without a Batch layer?
  4. What role does Tiered Storage in Apache Kafka (KIP-405) play in making Kappa Architecture practical?
  5. How does the Serving Layer in Lambda Architecture merge Batch Views and Speed Views at query time?
  6. How does Apache Flink support both Batch and Stream processing natively (Unified Processing)?
  7. What happens in Kappa Architecture when a bug is discovered in the stream processing code?
  8. Why did Jay Kreps propose Kappa Architecture as a replacement for Nathan Marz's Lambda Architecture?
  9. When is Lambda Architecture still used in enterprise data platforms today?
  10. How does Delta Lake / Apache Iceberg support Kappa Architecture on cloud blob storage?

7. Gold Standard Answers

  1. Lambda: Dual path (Batch Layer for batch views + Speed Layer for real-time views, merged at Serving Layer). Kappa: Single path (ALL data flows through Stream Layer, backfills executed by replaying event log).
  2. Lambda requires writing business logic TWICE (e.g. MapReduce/Spark for Batch, Storm/Flink for Speed). Differences in language or libraries cause Data Drift (Batch and Speed layers output conflicting answers).
  3. Replay: Store raw immutable events in Kafka/S3. When backfilling, spin up a second instance of the stream processing job, point its start offset to time zero (or historical timestamp), process events at max speed, and swap output views.
  4. Kafka Tiered Storage offloads cold log segments from local broker SSDs to cheap S3/GCS object storage. Enables keeping infinite historical event logs in Kafka cheaply, making Kappa backfills practical.
  5. Serving Layer queries both stores: Result = BatchView(t0 to t_nightly) + SpeedView(t_nightly to t_now). When the nightly batch job finishes, the old Speed View data is truncated.
  6. Flink treats Batch as a special case of Streaming: Bounded streams = Batch processing; Unbounded streams = Stream processing. The core runtime engine is identical.
  7. Bug fix in Kappa: (1) Fix code, (2) Deploy new job Version 2 reading from offset 0 into View V2, (3) Wait for V2 to catch up to real-time, (4) Switch Serving Layer API alias to View V2, (5) Terminate Version 1.
  8. Kreps argued that operating two separate codebases and processing engines (Lambda) was an operational nightmare, and modern stream processors (Flink/Samza) were powerful enough to handle both real-time and historical replay.
  9. Used in legacy environments where massive multi-terabyte batch computations (e.g., complex graph processing, matrix operations) cannot run efficiently on stream processing engines.
  10. Open table formats (Iceberg/Delta Lake) allow stream processors (Flink) to append real-time streams directly to cloud S3 data lakes with ACID guarantees, acting as an immutable event log for Kappa pipelines.

8. Common Mistakes

  • Building a Lambda Architecture in a new startup, duplicating processing logic across PySpark and Flink codebases.
  • Attempting Kappa Architecture with a 24-hour Kafka log retention policy (makes historical replaying impossible).
  • Forgetting to tier old Kafka log segments to cloud object storage (S3), resulting in massive broker SSD disk bills.
  • Overwriting production stream views directly during Kappa backfills instead of processing into a secondary V2 view before swapping.
  • Retaining Lambda Architecture purely out of habit when a single Flink / Spark Structured Streaming pipeline can handle both modes.

9. Keywords

Lambda Architecture, Kappa Architecture, Nathan Marz, Jay Kreps, Batch Layer, Speed Layer, Serving Layer, Code Duplication, Data Drift, Stream Replay, Offset Zero, Tiered Storage, KIP-405, Apache Flink, Apache Kafka.

10. Cheat Sheet

  • Core Difference: Lambda = Dual path (Batch Layer + Speed Layer merged at Serving Layer); Kappa = Single path (ALL data flows through 1 Stream Processing engine).
  • Lambda Flaw: Code Duplication & Data Drift (writing logic twice in Spark and Flink causes conflicting metrics).
  • Kappa Principle: Treat historical backfills as Streaming over Bounded Historical Event Logs (Replay Kafka from offset zero).
  • Kafka Tiered Storage: Offloads historical log segments from expensive broker SSDs to S3, enabling infinite retention for Kappa replay.
  • Kappa Deployment Rule: Deploy new stream job \(\rightarrow\) Replay offset 0 to new V2 table \(\rightarrow\) Catch up to real-time \(\rightarrow\) Atomic swap router alias to V2.
  • Golden Rule: Default to Kappa Architecture for all modern real-time data platform designs.

11. Related Decisions