Phase 1: Architecture, ZAB Consensus Protocol & Core Foundations¶

Audience: Principal Distributed Systems Architect & Consensus Protocol Specialist
Source References: zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/

1. Ensemble Peer Roles: Leader, Follower & Observer¶

A ZooKeeper cluster is called an Ensemble. Physical machines participate in one of three distinct roles:

┌────────────────────────────────────────────────────────────────────────┐
│ Leader (QuorumPeer: LEADING)                                           │
├────────────────────────────────────────────────────────────────────────┤
│ - Exactly ONE active leader per ensemble at any time                   │
│ - Accepts all write mutations; assigns globally monotonic zxid         │
│ - Coordinates ZAB atomic broadcast (Propose -> Quorum Ack -> Commit)   │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │ ZAB Protocol Stream
         ┌──────────────────────────┴──────────────────────────┐
         ▼                                                     ▼
┌────────────────────────────────────┐       ┌───────────────────────────────────┐
│ Follower (QuorumPeer: FOLLOWING)   │       │ Observer (QuorumPeer: OBSERVING)  │
├────────────────────────────────────┤       ├───────────────────────────────────┤
│ - Full voting participant          │       │ - Non-voting read replica         │
│ - Forwards client writes to Leader │       │ - Receives committed state from   │
│ - Votes in Leader Election &       │       │   Leader via INFORM packets       │
│   ZAB proposal quorums             │       │ - Scales read throughput without  │
│ - Replicates in-memory DataTree    │       │   increasing write quorum latency │
└────────────────────────────────────┘       └───────────────────────────────────┘

2. The ZAB Consensus Protocol (ZooKeeper Atomic Broadcast)¶

ZAB is a high-performance primary-backup protocol specifically engineered for tree-structured coordination.

ZAB Two-Phase Atomic Broadcast:
1. Client ──► Follower: setData("/config", "val2")
2. Follower ──► Leader: Forwards REQUEST packet

3. Leader initiates ZAB Proposal:
   - Increments low 32 bits of zxid: zxid = 0x200000004
   - Logs transaction to local FileTxnLog
   - Broadcasts PROPOSAL(0x200000004, setData) to all Followers

4. Followers validate and persist:
   - Append to local FileTxnLog (fsync)
   - Send ACK(0x200000004) back to Leader

5. Leader checks Quorum:
   - Once Leader receives ACKs from floor(N/2) + 1 nodes:
     - Applies mutation to local in-memory DataTree
     - Broadcasts COMMIT(0x200000004) to all Followers
     - Sends INFORM to Observers

6. Followers receive COMMIT:
   - Apply mutation to local in-memory DataTree
   - Return success to client!

2.1 FIFO Client Order & Total Order Invariants¶

ZAB enforces two strict mathematical invariants: 1. Total Order: If message $A$ is committed before message $B$ by one server, $A$ will be committed before $B$ by all servers. 2. Causal Consistency / Linearizable Writes: Writes are executed in the exact chronological sequence in which they were issued by the client application.

3. Cellular Layout of the 64-Bit `zxid`¶

Every transaction in ZooKeeper is identified by a 64-bit integer (zxid):

┌────────────────────────────────────────┬────────────────────────────────────────┐
│ Epoch Identifier (High 32 Bits)        │ Transaction Counter (Low 32 Bits)      │
│ Bits 32 to 63                          │ Bits 0 to 31                           │
└────────────────────────────────────────┴────────────────────────────────────────┘

Epoch: Represents the term of leadership. Each time a new leader is elected through Fast Leader Election, the epoch increments by 1.
Counter: Monotonically increments for every proposal proposed by that leader. It resets to 0 whenever a new epoch begins.
Comparison Rule: $$\text{zxid}_A > \text{zxid}_B \iff (\text{epoch}_A > \text{epoch}_B) \lor (\text{epoch}_A = \text{epoch}_B \land \text{counter}_A > \text{counter}_B)$$

4. Quorum Mathematics: Why Odd Ensemble Numbers?¶

ZooKeeper requires a strict majority to function: $$\text{Quorum Size} = \left\lfloor \frac{N}{2} \right\rfloor + 1$$

Total Nodes ($N$)	Quorum Needed	Maximum Failures Tolerated ($F$)
1	1	0
3	2	1
4	3	1 (Zero extra resilience over 3 nodes!)
5	3	2
6	4	2 (Zero extra resilience over 5 nodes!)
7	4	3

Why an Even Number of Nodes is an Anti-Pattern:¶

Adding a 4th node to a 3-node cluster does NOT increase fault tolerance (both tolerate only 1 failure). However, a 4-node cluster requires 3 nodes to agree on every write instead of 2, increasing network latency and reducing availability! Always deploy odd ensemble counts ($3, 5, 7$).