Skip to content

Master Connected Memory Map: The ZooKeeper Cellular Ecosystem

Audience: Principal Distributed Systems Architect & Consensus Protocol Specialist
Purpose: A comprehensive graphical and architectural memory map connecting every cellular component in Apache ZooKeeper—from client sessions and NIO channels to the in-memory DataTree, WatchManager, ZAB atomic broadcast, and physical transaction logs.

1. Executive Macro Ecosystem Map

graph TD
    classDef client fill:#e1f5fe,stroke:#0288d1,stroke-width:2px,color:#01579b;
    classDef peer fill:#ede7f6,stroke:#512da8,stroke-width:2px,color:#311b92;
    classDef mem fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#e65100;
    classDef disk fill:#fbe9e7,stroke:#d84315,stroke-width:2px,color:#bf360c;
    classDef zab fill:#e8f5e9,stroke:#388e3c,stroke-width:2px,color:#1b5e20;

    CLIENT["Client Application (Curator Framework)"]:::client

    subgraph ENSEMBLE ["ZooKeeper Distributed Quorum Ensemble (2F + 1 Nodes)"]
        subgraph LEADER_NODE ["Leader Peer Node (QuorumPeer: LEADING)"]
            LEADER_NIO["ServerCnxnFactory (Netty / NIO Socket Server)"]:::peer
            SESSION_TRACKER["SessionTrackerImpl (Expirations & Heartbeats)"]:::peer

            subgraph REQUEST_PROCESSOR_PIPELINE ["RequestProcessor Pipeline"]
                PREP_PROC["PrepRequestProcessor (Generates Txn Header & zxid)"]:::zab
                PROPOSAL_PROC["ProposalRequestProcessor (ZAB Proposal Broadcaster)"]:::zab
                COMMIT_PROC["CommitProcessor (Quorum Consensus Coordinator)"]:::zab
                TO_BE_APPLIED["ToBeAppliedRequestProcessor"]:::zab
                FINAL_PROC["FinalRequestProcessor (Applies Mutation to DataTree)"]:::zab
            end

            subgraph IN_MEMORY_DATABASE ["ZKDatabase (In-Memory Heap Engine)"]
                DATATREE["DataTree (Hierarchical ConcurrentHashMap of Znodes)"]:::mem
                WATCH_MGR["WatchManager (Path & Client Watch Registries)"]:::mem
            end

            subgraph DISK_PERSISTENCE ["Durable Block Persistence"]
                TXN_LOG["FileTxnLog (Append-Only log.* with 64MB Preallocation)"]:::disk
                SNAPSHOT["FileSnap (Fuzzy Periodic snapshot.* Dump)"]:::disk
            end
        end

        subgraph FOLLOWER_NODE ["Follower Peer Node (QuorumPeer: FOLLOWING)"]
            FOLLOWER_HANDLER["FollowerRequestProcessor (Forward Writes to Leader)"]:::peer
            SYNC_PROC["SyncRequestProcessor (Flushes TxnLog to Disk)"]:::disk
            ACK_SENDER["Send ACK to Leader"]:::zab
            FOLLOWER_TREE["Follower DataTree (In-Memory Replicated State)"]:::mem
        end

        OBSERVER_NODE["Observer Node (Non-Voting Read Scaler)"]:::peer
    end

    CLIENT -->|Session TCP Connect| LEADER_NIO
    LEADER_NIO --> SESSION_TRACKER
    LEADER_NIO --> PREP_PROC
    PREP_PROC --> PROPOSAL_PROC

    PROPOSAL_PROC -->|ZAB PROPOSAL message| FOLLOWER_HANDLER
    FOLLOWER_HANDLER --> SYNC_PROC
    SYNC_PROC --> ACK_SENDER
    ACK_SENDER -->|ZAB ACK message| COMMIT_PROC

    COMMIT_PROC -->|"Quorum Reached: Send ZAB COMMIT"| TO_BE_APPLIED
    TO_BE_APPLIED --> FINAL_PROC
    FINAL_PROC -->|Atomic Mutation| DATATREE
    FINAL_PROC -->|Trigger Matching Watches| WATCH_MGR
    WATCH_MGR -->|Fire Watch Event| LEADER_NIO
    LEADER_NIO -->|Deliver WatchedEvent| CLIENT

    PROPOSAL_PROC --> TXN_LOG
    DATATREE -.->|Periodic Background Dump| SNAPSHOT

2. Regional Deep-Zoom Maps

2.1 Regional Deep-Zoom A: In-Memory DataTree & DataNode Structure

How ZooKeeper represents the hierarchical filesystem in JVM heap memory:

graph TD
    classDef node fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px;
    classDef data fill:#fff9c4,stroke:#fbc02d,stroke-width:2px;
    classDef stat fill:#e3f2fd,stroke:#1565c0,stroke-width:2px;

    ROOT["Root Znode: / (DataNode)"]:::node
    SERVICES["/services (DataNode)"]:::node
    ORDERS["/services/orders (DataNode)"]:::node
    EPHEMERAL["/services/orders/lock-00000001 (EPHEMERAL_SEQUENTIAL)"]:::node

    STAT["StatPersisted (Metadata Header)"]:::stat
    DATA_BYTES["byte[] data (Payload <= 1MB)"]:::data
    CHILDREN["HashSet<String> children"]:::node

    ROOT --> SERVICES
    SERVICES --> ORDERS
    ORDERS --> EPHEMERAL

    EPHEMERAL --- STAT
    EPHEMERAL --- DATA_BYTES
    EPHEMERAL --- CHILDREN

    STAT --> CZXID["czxid: Creation zxid"]
    STAT --> MZXID["mzxid: Last Modified zxid"]
    STAT --> VERSION["version: Node Data Version Counter"]
    STAT --> EPHEMERAL_OWNER["ephemeralOwner: 64-bit SessionID (If Ephemeral)"]

2.2 Regional Deep-Zoom B: Fast Leader Election (FLE) Quorum Voting

graph TD
    classDef candidate fill:#ede7f6,stroke:#512da8,stroke-width:2px;
    classDef vote fill:#fff3e0,stroke:#f57c00,stroke-width:2px;
    classDef state fill:#e8f8f5,stroke:#117a65,stroke-width:2px;

    PEER_START["Node Enters State: LOOKING"]:::candidate
    PROPOSE_SELF["Create Vote: (my_epoch, my_last_zxid, my_id)"]:::vote
    BROADCAST["Broadcast Notification to all Quorum Peers"]:::vote

    COMPARE_RULE["Compare Received Vote (v) vs Local Vote (l)"]

    PEER_START --> PROPOSE_SELF
    PROPOSE_SELF --> BROADCAST
    BROADCAST --> COMPARE_RULE

    COMPARE_RULE -->|"Rule 1: v.epoch > l.epoch"| UPDATE_VOTE["Adopt Candidate's Vote!"]
    COMPARE_RULE -->|"Rule 2: v.epoch == l.epoch AND v.zxid > l.zxid"| UPDATE_VOTE
    COMPARE_RULE -->|"Rule 3: Epoch & zxid equal BUT v.myid > l.myid"| UPDATE_VOTE
    COMPARE_RULE -->|"Else: Keep Local Vote"| RE_BROADCAST["Re-broadcast Local Vote"]

    UPDATE_VOTE --> CHECK_QUORUM["Did Candidate receive votes from Majority (floor(N/2) + 1)?"]
    CHECK_QUORUM -->|"YES: Candidate becomes LEADER"| BECOME_LEADER["Transition: LEADING / FOLLOWING"]:::state

3. The 3 Primary Nervous System Pathways (In Motion)

Pathway 1: ZAB Atomic Broadcast Write Path

How a mutating write from a client is sequenced by the leader, acknowledged by a quorum of followers, and applied to the in-memory DataTree.

sequenceDiagram
    autonumber
    actor Client as "ZooKeeper Client (Curator)"
    participant Follower as "Follower Peer"
    participant Leader as "Leader Peer"
    participant LeaderTxnLog as "Leader FileTxnLog (NVMe)"
    participant FollowerSync as "Follower SyncRequestProcessor"
    participant DataTree as "In-Memory DataTree"

    Client->>Follower: create("/app/node", "payload", EPHEMERAL)
    Follower->>Leader: Forward Write Request (REQUEST message)

    Note over Leader: PrepRequestProcessor generates Header & monotonic zxid
    Leader->>LeaderTxnLog: Append transaction to local FileTxnLog
    Leader->>Follower: Broadcast ZAB PROPOSAL(zxid, txn)

    Follower->>FollowerSync: Log transaction to local FileTxnLog
    FollowerSync->>FollowerSync: Flush to disk (fsync)
    FollowerSync-->>Leader: Return ZAB ACK(zxid)

    Note over Leader: Quorum Reached (Leader + 1 Follower out of 3)!
    Leader->>DataTree: Apply mutation in FinalRequestProcessor
    Leader->>Follower: Broadcast ZAB COMMIT(zxid)
    Follower->>DataTree: Apply mutation to local DataTree

    Leader-->>Follower: Return SUCCESS response
    Follower-->>Client: Return WatchedEvent / Created Path

Pathway 2: Fast Leader Election (FLE) on Node Crash

How remaining nodes recover from an active leader death and achieve consensus.

sequenceDiagram
    autonumber
    participant Node1 as "Node 1 (Follower, zxid: 105)"
    participant Node2 as "Node 2 (Follower, zxid: 108)"
    participant Node3 as "Node 3 (Dead Leader)"

    Note over Node3: Leader Process Dies abruptly (Power Outage)!
    Node1->>Node1: Heartbeat timeout expires (tickTime * syncLimit)
    Node2->>Node2: Heartbeat timeout expires

    Node1->>Node1: Transition to LOOKING state; electionEpoch++
    Node2->>Node2: Transition to LOOKING state; electionEpoch++

    Node1->>Node2: Vote(epoch: 2, zxid: 105, myid: 1)
    Node2->>Node1: Vote(epoch: 2, zxid: 108, myid: 2)

    Note over Node1: Node 1 evaluates Node 2 Vote: zxid 108 > 105!
    Node1->>Node1: Changes local vote to (epoch: 2, zxid: 108, myid: 2)
    Node1->>Node2: Re-broadcast updated Vote

    Note over Node2: Node 2 observes majority (Node 1 + Node 2 = 2/3 votes for Node 2)!
    Node2->>Node2: State = LEADING (Elected Leader)
    Node1->>Node1: State = FOLLOWING (Attaches to Node 2)

Pathway 3: One-Time Watch Registration & Event Dispatch

How watch triggers notify clients and require immediate re-registration.

sequenceDiagram
    autonumber
    actor Client as "Application Client (Curator)"
    participant Server as "ZooKeeper Server (ServerCnxn)"
    participant WatchMgr as "WatchManager (In-Memory Map)"
    participant DataTree as "DataTree Engine"

    Client->>Server: getData("/app/config", watch=true)
    Server->>WatchMgr: Register (path="/app/config", cnxn=ClientSocket)
    Server-->>Client: Return current config payload { "db": "pg1" }

    Note over Client,Server: Client is now actively watching /app/config

    actor Admin as "Admin Operator"
    Admin->>Server: setData("/app/config", "{ 'db': 'pg2' }")
    Server->>DataTree: Mutate znode data in RAM
    Server->>WatchMgr: TriggerAndRemove("/app/config", EventType.NodeDataChanged)
    Note over WatchMgr: Watch is REMOVED from memory! (One-Time Trigger!)

    WatchMgr->>Server: Serializes WatchedEvent
    Server->>Client: Send WatchedEvent: NodeDataChanged on /app/config

    Note over Client: Client MUST immediately issue getData(watch=true) to re-register!

4. Cause-and-Effect Contention & Pathology Matrix

Root Event / Pathology Primary Cellular Symptom Immediate Downstream Blast Radius Emergency Diagnostic & Recovery Action
Watch Storm (Herd Effect) 10,000 clients watch a single znode (e.g. /lock) Network NIC Saturation: A single update triggers 10,000 concurrent socket write events; server freezes. Redesign lock recipe to watch only the immediately preceding sequential node (/lock/x-00002 watches /lock/x-00001).
TxnLog Disk Write Stall Transaction log shares disk with active OS / logs Quorum Drops Follower: fsync() blocks \(> \text{tickTime} \times \text{syncLimit}\); leader convicts follower as dead. Relocate dataLogDir to a dedicated physical NVMe SSD isolated from snapshots and OS logs.
Ephemeral Node Accumulation Client disconnects without calling close() Stale Locks Retained: Ephemeral znodes remain alive until sessionTimeout (default 20–40 seconds) expires. Tune minSessionTimeout and maxSessionTimeout; ensure client cleanly terminates sessions.
Large Znode Payloads (> 1MB) Storing multi-megabyte JSON configs in a znode JVM Heap Exhaustion: High memory serialization pressure; exceeds jute.maxbuffer limit. ZooKeeper is a coordination service, NOT a database! Keep znode payloads \(< 4\text{KB}\); store large data in S3/Postgres.
Fuzzy Snapshot Compaction Stall Snapshot generation (autopurge) disabled Disk Space Exhaustion: Millions of historical snapshot.* and log.* files fill physical disk. Configure autopurge.purgeInterval = 1 and autopurge.snapRetainCount = 5 in zoo.cfg.

5. Rapid Revision Flashcards

  1. What is the structure of a 64-bit zxid (ZooKeeper Transaction ID)?
  2. High 32 bits: Epoch (Increments each time a new leader is elected).

  3. Low 32 bits: Monotonic Counter (Increments on every transaction proposal; resets to 0 on new epoch).

  4. Why does ZooKeeper recommend an ODD number of ensemble nodes (3, 5, 7)?
    Because quorum requires a strict majority: \(\lfloor N/2 \rfloor + 1\). A 5-node cluster survives 2 failures (\(\lfloor 5/2 \rfloor + 1 = 3\)). A 6-node cluster also survives only 2 failures (\(\lfloor 6/2 \rfloor + 1 = 4\)), while requiring more network consensus overhead. Even numbers provide zero extra fault tolerance!

  5. What is a "Fuzzy Snapshot" in ZooKeeper?
    When taking a snapshot, ZooKeeper does NOT lock the in-memory DataTree. It streams the tree to disk while concurrent mutations continue executing. The snapshot is point-in-time inconsistent ("fuzzy"). Consistency is restored during reboot by replaying the subsequent transactions from the append-only FileTxnLog.

  6. Why are ZooKeeper Watches "One-Time" Triggers?
    To prevent slow clients from falling behind an infinite stream of updates. Once a watch fires, it is deleted from the server's memory. The client must explicitly re-register the watch to observe subsequent changes. (Note: Persistent recursive watches were added in v3.6+ for tree monitoring).

  7. Why did Apache Kafka migrate from ZooKeeper to KRaft (KIP-500)?
    At 200,000+ topic partitions, Kafka's metadata exceeded ZooKeeper's memory and watch capacity. Changes triggered massive Watch Storms, and controller failover took minutes to reload the full DataTree into RAM. KRaft integrated an internal Raft metadata quorum directly inside Kafka brokers, enabling millions of partitions with sub-second failover.