Skip to content

Kafka vs Pulsar

1. Decision

Choose Apache Kafka for battle-tested industry standard event streaming, massive ecosystem integration (Kafka Connect, Kafka Streams, Flink), and operational simplicity with monolithic storage/compute broker nodes; choose Apache Pulsar for multi-tenant enterprise messaging requiring unified Queue + Stream semantics, decoupled compute/storage architecture (BookKeeper), seamless tiered storage to S3, and instant partition expansion.

2. 30-Second Answer

Apache Kafka and Apache Pulsar are both high-performance, distributed event streaming platforms, but feature fundamentally different architectural designs. Kafka uses a monolithic architecture where broker nodes manage both message serving (compute) and log storage on locally attached disks. This delivers maximum sequential disk I/O performance, but makes partition rebalancing, cluster scaling, and cold historical data storage operationally heavy. Apache Pulsar uses a decoupled, multi-tier architecture separating stateless Brokers (compute) from Apache BookKeeper (storage nodes). Pulsar supports native Multi-Tenancy, tiered storage to S3/GCS, dynamic partition expansion without data re-replication, and unified messaging subscriptions (Queueing + Streaming).

3. Decision Matrix

Option Best For Avoid When Latency Scalability Consistency Operational Complexity
Apache Kafka Standard event streaming, CDC pipelines, Kafka Streams / Flink analytics, vast open-source ecosystem Multi-tenant isolation requirements, frequent cluster rebalancing, immediate zero-replication scale-out Low (<10ms batch throughput optimized) High (Scales via topic partition sharding across broker disks) Strict partition order; Configurable acks=all Medium (KRaft mode simplifies management)
Apache Pulsar Multi-tenant cloud platforms, unified Queue + Stream messaging, tiered S3 archiving, instant node expansion Simple architectures where running 3 separate distributed systems (Brokers, BookKeeper, ZooKeeper) is overkill Very Low (<5ms p99 write latency via BookKeeper journal) Extremely High (Decoupled compute & storage scale independently) Strong consistency via BookKeeper Quorums (\(AckQuorum\)) High (Requires managing Brokers, BookKeeper, and ZooKeeper)

4. Selection Rules

Requirement Best Choice Rationale / Example
Vast Open-Source Ecosystem & Tooling Kafka Universal support across Kafka Connect, Flink, Spark, Debezium, and managed cloud offerings
Unified Queueing + Event Streaming Pulsar Supports Exclusive, Failover, Shared (Queue), and Key_Shared consumer subscriptions
Native Multi-Tenancy & Isolation Pulsar Built-in Tenant/Namespace hierarchies with ACLs, quotas, and isolation policies
Automated Tiered Storage to AWS S3 Pulsar Offloads historical log segments automatically to cheap cloud object storage
Instant Node Scaling Without Data Rebalancing Pulsar Adding stateless Brokers or Bookies expands capacity without moving historical data
Simpler Deployment Architecture (KRaft) Kafka KRaft mode eliminates external metadata clusters; run pure Kafka broker nodes

5. Tradeoffs

  1. Monolithic Compute/Storage (Kafka) vs Decoupled Storage Architecture (Pulsar)
  2. Gain (Kafka): Simpler operational footprint; direct sequential disk I/O from broker OS Page Cache.
  3. Lose (Kafka): Scaling storage requires adding compute nodes; adding nodes triggers heavy partition re-replication data moves across the network.
  4. Unified Subscriptions (Pulsar) vs Pure Partition Streaming (Kafka)
  5. Gain (Pulsar): Use Shared subscription mode for competing worker queues and Failover mode for streaming logs on the same topic.
  6. Lose (Pulsar): Higher operational complexity managing BookKeeper ledger quorum configurations.
  7. Automated Tiered Storage (Pulsar) vs Local Disk Retention (Kafka)
  8. Gain (Pulsar): Retain years of historical topics seamlessly by offloading old ledgers to S3/GCS automatically.
  9. Lose (Pulsar): Requires managing S3 offloader drivers and BookKeeper ledger metadata transitions.
  10. Ecosystem Ubiquity vs Feature Sophistication
  11. Gain (Kafka): Massive community, extensive stack overflow solutions, ready-to-use connectors for every database/service.
  12. Lose (Kafka): Operational toil during large cluster expansion and partition rebalancing (mitigated by Cruise Control).
  13. Operational Infrastructure Footprint
  14. Gain (Kafka): Single binary daemon process (KRaft mode).
  15. Lose (Pulsar): Must deploy, tune, and monitor three separate distributed systems: Pulsar Brokers, BookKeeper Bookies, and ZooKeeper.

6. Interview Counter Questions

  1. How does Apache Pulsar's two-layer architecture (Brokers vs BookKeeper Bookies) differ from Kafka's single-layer architecture?
  2. Why is partition rebalancing heavy in Kafka but instant in Pulsar?
  3. How do Pulsar's 4 Subscription Modes (Exclusive, Failover, Shared, Key_Shared) work?
  4. How does Apache BookKeeper achieve low latency writes using Quorum Ledger writing (\(EnsembleSize\), \(WriteQuorum\), \(AckQuorum\))?
  5. What is Tiered Storage in Pulsar and how does it reduce long-term log storage costs?
  6. How does Kafka KRaft mode compare to Pulsar's metadata dependencies?
  7. Why is Pulsar considered better suited for Multi-Tenant SaaS platforms than Kafka?
  8. How does Geo-Replication differ between Kafka (MirrorMaker 2) and Pulsar (Native Geo-Replication)?
  9. What is Pulsar Functions and how does it compare to Kafka Streams / KSQLDB?
  10. Which system provides lower p99 write tail latencies and why?

7. Gold Standard Answers

  1. Kafka brokers serve requests and store log segments on local attached disks. Pulsar brokers are stateless compute nodes that handle client connections; actual log segments are written to Apache BookKeeper (Bookie storage nodes) over the network.
  2. In Kafka, adding a broker node requires copying gigabytes of partition segment files across the network (re-replication). In Pulsar, new Bookkeeper nodes immediately accept new log ledgers without moving historical data.
  3. Exclusive: 1 consumer per topic. Failover: Active/Standby consumers. Shared: Competing worker queue distribution. Key_Shared: Competing worker distribution while guaranteeing ordering per message key.
  4. BookKeeper writes log entries to an ensemble of Bookies (\(E\)). \(WriteQuorum\) (\(W\)) is how many Bookies store copies. \(AckQuorum\) (\(A\)) is how many acknowledgments the broker waits for before returning success to the client (e.g., \(E=3, W=3, A=2\)).
  5. Pulsar automatically moves immutable closed log ledgers from expensive BookKeeper SSD storage to cheap cloud object storage (AWS S3 / GCS) while keeping the data seamlessly queryable via standard topic offsets.
  6. Kafka KRaft mode uses internal Raft consensus to manage metadata without ZooKeeper, running a single unified node binary. Pulsar still depends on ZooKeeper (or etcd) for cluster coordination alongside BookKeeper and Brokers.
  7. Pulsar built native multi-tenancy from day one (tenant/namespace/topic structure), supporting storage quotas, rate-limiting, and isolation policies natively per tenant. Kafka requires custom ACLs and external topic naming conventions.
  8. Pulsar has native multi-datacenter asynchronous Geo-Replication configured at the namespace level without external tools. Kafka requires deploying and operating external MirrorMaker 2 or Confluent Replicator clusters.
  9. Pulsar Functions is a lightweight serverless compute framework built directly inside Pulsar brokers for simple transformations (map, filter). Kafka Streams is a heavy Java client library for complex stateful stream processing.
  10. Pulsar generally delivers lower p99 write tail latencies because BookKeeper writes to a dedicated, low-latency Journal disk for fast sync, while asynchronously writing to Ledger disks for read performance.

8. Common Mistakes

  • Underestimating the operational complexity of deploying Pulsar (running Brokers + BookKeeper + ZooKeeper simultaneously).
  • Assuming Kafka cannot scale partitions (Kafka scales easily; adding nodes requires partition rebalancing via Cruise Control).
  • Choosing Pulsar for simple streaming applications without needing multi-tenancy or tiered storage, adding unnecessary operational overhead.
  • Failing to tune BookKeeper \(WriteQuorum\) and \(AckQuorum\) settings, causing unexpected latency or durability drops.
  • Using Kafka for multi-tenant SaaS platforms where complete tenant data and rate-limit isolation are required out-of-the-box.

9. Keywords

Kafka, Pulsar, BookKeeper, Bookie, Stateless Broker, Decoupled Architecture, Multi-Tenancy, Tiered Storage, Ledger, Ensemble, WriteQuorum, AckQuorum, Key_Shared, KRaft, MirrorMaker 2, Pulsar Functions.

10. Cheat Sheet

  • Architecture: Kafka = Monolithic (Compute + Storage on Broker); Pulsar = Decoupled (Stateless Brokers + BookKeeper storage).
  • Cluster Scaling: Kafka requires moving partition data to new nodes (re-replication); Pulsar adds Bookkeeper nodes instantly without moving historical data.
  • Messaging Model: Kafka = Pure partitioned log stream; Pulsar = Unified Queue (Shared) + Event Stream (Failover/Key_Shared).
  • Tiered Storage: Pulsar offloads old log ledgers to S3 automatically; Kafka requires enterprise plugins or manual archiving.
  • Ecosystem: Kafka has 10x larger community, connector library (Kafka Connect), and stream processing adoption (Flink/Spark).
  • Multi-Tenancy: Pulsar has native tenant/namespace/topic isolation built-in.

11. Related Decisions