Skip to content

Offsets

1. What is it? (2-3 lines)

An Offset is a monotonically increasing 64-bit integer identifier assigned to each message sequentially as it arrives within a stream partition (e.g. Apache Kafka), representing the message's exact logical position in the partition log.

2. Why does it exist?

Traditional message queues (like RabbitMQ) delete messages immediately after consumer acknowledgment, making historical replaying impossible and coupling message retention to consumer execution speed. Offsets treat topic partitions as immutable append-only logs on disk, decoupling message storage from consumer position tracking.

3. Where is it used?

  • Distributed Event Logs: Apache Kafka (__consumer_offsets), Redpanda, Apache Pulsar.
  • Change Data Capture (CDC): Debezium / Postgres WAL LSN numbers.
  • Stream Processing Engines: Apache Flink, Spark Structured Streaming (checkpointing consumer offsets).

4. How does it work? (simple flow)

  1. Append & Assignment: Producer sends message to Partition 0 \(\to\) Broker appends message to end of log file \(\to\) Assigns Offset = 1042.
  2. Consumer Fetch: Consumer requests messages starting at Offset = 1042.
  3. Processing & Commit: Consumer processes the message batch locally \(\to\) Sends an offset commit request (Offset = 1043) back to broker.
  4. Offset Storage: Broker stores committed offsets in the internal __consumer_offsets topic mapped to (group_id, topic, partition).
  5. Consumer Crash / Restart: If consumer worker crashes, replacement consumer reads last committed offset (1043) from __consumer_offsets and resumes consumption seamlessly without missing messages.

5. Advantages

  • Enables historical message replaying (re-seeking consumer offset to offset 0 to re-process historical data).
  • Decouples message retention from consumption speed (multiple consumer groups track independent offsets over the same log).
  • Precise metric calculation: Consumer Lag \(= \text{Log End Offset (LEO)} - \text{Committed Offset}\).

6. Limitations

  • At-Least-Once Duplicate Delivery: If consumer crashes after processing business logic but before committing offset, replacement consumer re-processes messages (requires Idempotency!).
  • Manual offset management complexity (enable.auto.commit = false) when handling asynchronous worker threads.

7. Common interview questions (10)

  1. What is the difference between Log End Offset (LEO), High Watermark (HW), and Committed Offset in Kafka?
  2. How does enable.auto.commit = true vs manual offset committing (commitSync / commitAsync) impact message delivery guarantees?
  3. What is Consumer Lag and how do you monitor it in production Grafana dashboards?
  4. How do you rewind a Consumer Group's offset to 24 hours ago to re-process historical data after a bug fix?
  5. What causes duplicate message processing when committing offsets asynchronously?
  6. How does Kafka store committed offsets in the internal __consumer_offsets topic?
  7. What happens if a consumer commits an offset before the message batch has completed processing?
  8. How do transactional producers and consumers achieve Exactly-Once semantics using atomic offset commits?
  9. What is auto.offset.reset (earliest vs latest vs none) and when is each triggered?
  10. How does Apache Flink manage Kafka offsets inside its stateful checkpointing snapshots?

8. Common misconceptions

  • Misconception: Committing offset 100 means you have committed message 100. Fact: Committing offset 100 indicates the consumer expects to read message 101 next (all messages up to 100 are processed).
  • Misconception: Offsets are unique across an entire Kafka topic. Fact: Offsets are unique strictly within a single partition.

10. 30-second cheat sheet

  • Definition: Sequential 64-bit integer index representing message position in a partition log.
  • Consumer Lag Formula: \(\text{Lag} = \text{Log End Offset (LEO)} - \text{Committed Offset}\).
  • Storage: Stored in __consumer_offsets topic per (group_id, topic, partition).
  • Commit Best Practice: Use Manual Commit (enable.auto.commit = false) after processing message business logic.
  • Offset Reset: earliest = Start from beginning; latest = Start from newest messages.