Skip to content

Consumer Groups

1. What is it? (2-3 lines)

A Consumer Group is a group of subscriber worker instances that pool together to consume, process, and load-balance messages from partitions of a message streaming topic (e.g., Apache Kafka), enabling parallel scale-out message consumption.

2. Why does it exist?

A single consumer thread cannot process high-velocity message streams (e.g. 100,000 msgs/sec). Consumer Groups provide a dynamic scale-out mechanism: each consumer instance in the group is assigned a mutually exclusive subset of topic partitions. If consumption lag spikes, adding new consumer instances scales throughput automatically without data duplication.

3. Where is it used?

  • Event Streaming Brokers: Apache Kafka, Apache Pulsar, Redpanda.
  • Message Queues: RabbitMQ (Competing Consumers pattern), AWS SQS (Multiple worker instances).

4. How does it work? (simple flow)

  1. Group Registration: 3 worker pods register with Kafka under the same group.id = "payment-processors".
  2. Partition Assignment: Group Coordinator (Kafka broker) assigns topic partitions to group instances.
  3. Example: Topic with 6 partitions \(\implies\) Worker 1 gets \(P_0, P_1\); Worker 2 gets \(P_2, P_3\); Worker 3 gets \(P_4, P_5\).
  4. Exclusive Consumption: Each partition is consumed by EXACTLY ONE consumer instance within the same group.
  5. Rebalance Trigger: If Worker 3 crashes (heartbeat timeout) or Worker 4 is added, the Group Coordinator triggers a Rebalance, reassigning partitions among active instances.

5. Advantages

  • Parallel, horizontal scale-out message processing across multiple worker nodes.
  • Fault Tolerance: Automated partition re-assignment (rebalancing) when a consumer pod crashes.
  • Broadcast + Queue Hybrid: Multiple distinct consumer groups (billing-group, analytics-group) consume the same topic independently without interfering with each other.

6. Limitations

  • Max Parallelism Cap: Maximum active consumers in a group is capped by the number of topic Partitions (\(N_{\text{consumers}} \le N_{\text{partitions}}\)). Extra consumers sit idle!
  • Rebalance Pause (Stop-The-World): Frequent consumer group rebalances pause message consumption across the group (mitigated by Cooperative Sticky Assignors).

7. Common interview questions (10)

  1. What happens if you add 10 consumer instances to a consumer group reading from a topic with 4 partitions?
  2. How does the Kafka Group Coordinator detect a dead consumer instance (Heartbeat thread vs Processing thread)?
  3. What is a Kafka Rebalance and how does Cooperative Sticky Assignor eliminate Stop-The-World pauses?
  4. How do two independent Consumer Groups (group-A and group-B) interact when reading the same Kafka topic?
  5. How does max.poll.interval.ms differ from session.timeout.ms in Kafka consumers?
  6. What is Static Group Membership in Kafka and how does it prevent rebalances during pod restarts?
  7. How does a Consumer Group maintain offset tracking in the __consumer_offsets internal topic?
  8. How do you scale message consumption when adding topic partitions is not allowed?
  9. What causes a Consumer Group to enter a continuous rebalance loop?
  10. How does RabbitMQ's Competing Consumers pattern differ from Kafka Consumer Groups?

8. Common misconceptions

  • Misconception: Multiple consumers in the same group can read the same partition simultaneously. Fact: A single partition is assigned to strictly ONE consumer within a given consumer group to preserve message ordering.
  • Misconception: Adding more consumers always increases throughput. Fact: Once consumer_count == partition_count, adding extra consumers adds zero throughput (extra consumers sit completely idle).

10. 30-second cheat sheet

  • Purpose: Pool worker instances (group.id) to process topic partitions in parallel.
  • Cardinality Constraint: Max Active Consumers = Number of Topic Partitions.
  • Partition Rule: 1 Partition \(\to\) 1 Consumer within the same group.
  • Multiple Groups: Different group.ids read the same topic independently (Pub/Sub pattern).
  • Pro Tip: Use Static Membership (group.instance.id) to prevent rebalance storms during Kubernetes pod rollouts.