Skip to content

Partitions

1. What is it? (2-3 lines)

A Partition is a fundamental unit of parallelism and storage in distributed message streaming platforms (e.g. Apache Kafka), representing an append-only, ordered log file of messages stored on disk within a larger logical topic.

2. Why does it exist?

A single message broker server cannot scale storage or processing throughput past its single-disk and single-NIC hardware limits. Partitions break a single logical Topic into multiple physical logs scattered across different broker nodes in a cluster, enabling massive horizontal write scaling and parallel consumer processing.

3. Where is it used?

  • Distributed Event Streams: Apache Kafka, Redpanda, Apache Pulsar (Partitioned Topics).
  • Distributed Databases: Shards/Partitions in DynamoDB, Cassandra, Elasticsearch indices.

4. How does it work? (simple flow)

  1. Producer Partitioning:
  2. Keyed Message: Producer hashes message key: Partition = murmur2(message_key) % NumPartitions. All messages with the same key (e.g., user_id = 42) land in the EXACT SAME PARTITION, guaranteeing strict message ordering!
  3. Keyless Message: Distributed via Sticky Partitioner / Round-Robin across available partitions.
  4. Broker Storage: Each partition maps to an on-disk directory containing segment files (.log, .index, .timeindex).
  5. Replication: Each partition has 1 Leader Replica (handles all writes/reads) and \(M\) Follower Replicas (in ISR - In-Sync Replicas list).

5. Advantages

  • Horizontal scale-out: Distributes a single multi-terabyte topic across dozens of physical broker nodes.
  • High Parallelism: Enables multiple consumers in a group to read different partitions concurrently.
  • Strict Order Guarantee: Guarantees strict sequential message ordering within a partition.

6. Limitations

  • Fixed Partition Bottleneck: Increasing partition count later is non-trivial (remaps message keys!).
  • Hot Partitions: Poorly chosen partition keys (e.g. static tenant key) route 90% of traffic to 1 single partition node.
  • High partition counts (e.g. >10,000 per broker) increase ZooKeeper/KRaft controller metadata overhead and failover latencies.

7. Common interview questions (10)

  1. How does message partitioning enable both high write throughput AND strict message ordering in Kafka?
  2. What happens to message key routing when you increase the partition count of an existing Kafka topic?
  3. How does the default Sticky Partitioner in modern Kafka producers optimize batching for keyless messages?
  4. What is a Hot Partition and how do you resolve partition skew caused by high-cardinality keys?
  5. What is the Relationship between Partition Count and Consumer Group scaling limits?
  6. How does Kafka replicate partition data across Lead and Follower replicas (ISR - In-Sync Replicas)?
  7. How do min.insync.replicas and acks=all interact at the partition level to guarantee zero data loss?
  8. How many partitions should a production Kafka topic have (Rules of Thumb for sizing)?
  9. What happens to a partition's availability if the Leader replica crashes and no Follower is in the ISR list?
  10. How do segment files (.log) work inside a physical partition directory on disk?

8. Common misconceptions

  • Misconception: Kafka guarantees global message ordering across an entire topic. Fact: Kafka guarantees strict message ordering strictly within a single partition.
  • Misconception: More partitions are always better. Fact: Excessive partitions inflate memory usage, increase open file descriptor counts, and slow down leader failover recovery times.

10. 30-second cheat sheet

  • Definition: Physical append-only log file on disk. Core unit of horizontal scaling & parallelism.
  • Key Routing: murmur2(key) % NumPartitions \(\implies\) Same key ALWAYS goes to same partition.
  • Ordering Guarantee: Strict order within a partition ONLY. Zero global topic ordering.
  • Parallelism Rule: Topic Partitions = Max Active Consumers in a Consumer Group.
  • Data Loss Defense: acks=all + min.insync.replicas=2 + Replication Factor = 3.