Skip to content

Kafka vs SQS

1. Decision

Choose AWS SQS for zero-operational-overhead, serverless, pay-per-message background task queuing and microservice decoupling on AWS; choose Apache Kafka for high-throughput event streaming, log-based event sourcing, message replayability, strict partition ordering, and real-time stream processing analytics.

2. 30-Second Answer

AWS Simple Queue Service (SQS) is a fully managed, serverless, point-to-point message queuing service provided by AWS. SQS requires zero cluster management, scales automatically from zero to millions of messages, and operates on a pay-per-request pricing model ($0.40 per million requests). Messages are transient: competing worker nodes pull tasks, process them, and delete them upon completion. Apache Kafka is a distributed event streaming platform built on an immutable, partitioned append-only log. Kafka requires running server broker clusters (self-hosted or MSK), but provides multi-consumer log retention, event replayability, microsecond stream processing, and sub-millisecond local network throughput. Choose SQS for serverless cloud task queues; choose Kafka for event-driven streaming architectures.

3. Decision Matrix

Option Best For Avoid When Latency Scalability Consistency Operational Complexity
AWS SQS (Standard / FIFO) Serverless AWS microservices, background job worker queues, unpredictable bursty task spikes, zero ops management High-volume log retention, message replayability, real-time stream analytics, multi-gigabyte ingestion Low (20-50ms API network RTT) Fully Managed Serverless (Virtually unlimited throughput auto-scaling) Transient; Deleted on worker deletion ACK Zero (Fully Managed Serverless Cloud)
Apache Kafka (Self-Hosted / MSK) High-velocity telemetry, CDC streaming (Debezium), event sourcing, stream processing (Flink), multi-consumer log analytics Simple serverless task queues with low/intermittent traffic volume Ultra-Low (<10ms batch throughput optimized) High (Linear scale-out via partition topic sharding) Durable immutable log; Strict partition ordering Medium to High (Broker cluster / MSK provisioning)

4. Selection Rules

Requirement Best Choice Rationale / Example
Zero Operational Overhead (Serverless) AWS SQS No brokers to manage, no partition rebalancing, automatic capacity auto-scaling
Event Replayability & Log Retention Kafka Retain events for 30+ days; rewind consumer group offsets to re-process historical events
Ultra-High Throughput Ingestion (>100k msg/sec) Kafka Sequential disk log appends and Linux Zero-Copy delivery outperform HTTP API queuing
AWS Lambda Serverless Event Triggers AWS SQS SQS triggers AWS Lambda functions natively with automated concurrency scaling
Pay-Per-Use Pricing for Bursty Traffic AWS SQS Scale to zero cost when idle; pay strictly per million SQS API requests
Strict Ordering Across Partition Keys Kafka Partition keys (user_id) guarantee sequence ordering across high-volume event streams

5. Tradeoffs

  1. Fully Managed Serverless vs Cluster Operations Management
  2. Gain (SQS): Zero infrastructure maintenance; AWS manages availability, durability, and scaling.
  3. Lose (SQS): Cloud vendor lock-in to AWS; API network request costs scale with volume.
  4. Pay-Per-Request Pricing vs Fixed Broker Instance Provisioning
  5. Gain (SQS): Cheaper for low-to-medium or bursty intermittent workloads (scales to zero).
  6. Lose (SQS): Continuous high-throughput workloads (>100,000 QPS) become drastically more expensive on SQS than Kafka instances.
  7. Transient Task Deletion vs Immutable Log Replayability
  8. Gain (SQS): Automatic queue cleanup; messages deleted immediately upon worker ACK.
  9. Lose (SQS): Cannot replay historical messages to new consumer applications.
  10. Competing Consumer Load Balancing vs Partition-Bounded Concurrency
  11. Gain (SQS): Dynamically add 500 Lambda workers to process a backed-up queue without partition limits.
  12. Lose (SQS): Standard SQS does not guarantee message delivery ordering.
  13. Visibility Timeout vs Consumer Group Offset Commits
  14. Gain (SQS): If a worker crashes during processing, the message automatically reappears in the queue after Visibility Timeout.
  15. Lose (SQS): Worker must process and delete within visibility window or duplicate executions occur.

6. Interview Counter Questions

  1. What is the difference between AWS SQS Standard Queues and SQS FIFO Queues?
  2. How does SQS Visibility Timeout work and what happens if a worker exceeds it?
  3. How do you implement Fanout messaging using AWS SNS + SQS?
  4. What is SQS Long Polling (WaitTimeSeconds=20) and why should you always enable it?
  5. Why is sustained 200,000 msg/sec streaming significantly cheaper on Kafka (AWS MSK) than SQS?
  6. How does message deduplication work in SQS FIFO Queues (MessageDeduplicationId)?
  7. What happens when SQS message consumption fails repeatedly (SQS Dead-Letter Queues)?
  8. How does AWS Lambda integrate natively with SQS vs Kafka as an event source?
  9. When would you choose AWS Kinesis Data Streams over Apache Kafka and AWS SQS?
  10. How do you handle poison-pill messages in AWS SQS?

7. Gold Standard Answers

  1. Standard: At-least-once delivery, best-effort ordering, nearly unlimited throughput. FIFO: Exactly-once processing, strict FIFO message ordering per Message Group ID, throughput capped at 300 msg/sec (or 3,000 with batching).
  2. Visibility Timeout hides a pulled message from other workers. If the worker fails to delete the message before the timeout expires (e.g., 30s), SQS makes the message visible again for another worker to process.
  3. SNS (Pub/Sub) fans out a single published message to multiple SQS queues (competing worker task pools), allowing independent microservices to consume a dedicated copy of the event asynchronously.
  4. Long Polling (WaitTimeSeconds=20) holds the ReceiveMessage API call open for up to 20s until a message arrives, eliminating empty API calls, reducing cost, and lowering latency.
  5. SQS charges \(0.40 per million API requests. At 200,000 QPS continuous write + read, SQS generates millions of API calls daily (~\)15,000+/mo). Kafka running on provisioned EC2/MSK instances amortizes infrastructure cost independently of request volume.
  6. SQS FIFO uses MessageDeduplicationId or content-based SHA-256 hashing to reject duplicate messages published within a 5-minute deduplication interval window.
  7. SQS tracks ApproximateReceiveCount. When a message is received but not deleted \(N\) times (maxReceiveCount), SQS automatically moves it to a configured Dead-Letter Queue (DLQ).
  8. SQS \(\rightarrow\) Lambda: AWS manages event polling and scales Lambda instances up to ReservedConcurrentExecutions. Kafka \(\rightarrow\) Lambda: AWS Lambda Event Source Mapping polls Kafka partitions, scaling 1 Lambda container per partition.
  9. AWS Kinesis is a fully managed cloud alternative to Kafka (partitioned streams, log retention). Choose Kinesis for native AWS streaming without managing Kafka brokers; choose SQS for simple task queues; choose Kafka for open-source ecosystem & higher throughput.
  10. Set maxReceiveCount (e.g., 3 retries). When a poison pill repeatedly crashes the Lambda/worker, SQS moves it to the SQS DLQ, unblocking the main task queue automatically.

8. Common Mistakes

  • Using SQS Standard Queues when strict FIFO message processing order is required.
  • Short polling SQS (WaitTimeSeconds=0), burning thousands of dollars on empty API requests.
  • Setting SQS Visibility Timeout shorter than worker execution time, causing duplicate parallel processing.
  • Choosing Kafka over SQS for simple, low-volume background tasks, adding immense operational broker management overhead.
  • Attempting to implement multi-consumer event replayability using AWS SQS.

9. Keywords

AWS SQS, SQS Standard, SQS FIFO, Kafka, MSK, Visibility Timeout, SNS + SQS Fanout, Long Polling, WaitTimeSeconds, MessageDeduplicationId, MessageGroupId, Dead-Letter Queue (DLQ), AWS Kinesis, Serverless.

10. Cheat Sheet

  • Core Difference: SQS = Fully managed serverless task queue (AWS native, transient task deletion); Kafka = Distributed event streaming log (Multi-consumer replay, high throughput).
  • SQS Queues: Standard = Unlimited scale, at-least-once, best-effort order; FIFO = 300 msg/sec limit, strict order per Message Group ID.
  • Cost Rule: SQS is cheaper for low/bursty traffic (scales to zero); Kafka is far cheaper for continuous high-throughput streams (>100k QPS).
  • Long Polling: Always set SQS WaitTimeSeconds=20 to eliminate empty API request billings.
  • Visibility Timeout: Ensure SQS Visibility Timeout is set longer than maximum worker processing time.
  • Fanout Pattern: Combine AWS SNS (Pub/Sub) + SQS (Queues) to broadcast single events to multiple queue consumers.

11. Related Decisions