Phase 3: Watch Mechanisms & Event Dispatching Architecture¶

Audience: Principal Distributed Systems Architect & High-Throughput Systems Programmer
Source References: zookeeper-server/src/main/java/org/apache/zookeeper/server/watch/

1. One-Time Watch Semantics & The Re-registration Race¶

A Watch is a one-time trigger placed on a znode by a client. When the znode changes, the server sends an asynchronous event to the client.

┌────────────────────────────────────────────────────────────────────────┐
│ The Classical One-Time Trigger Axiom                                   │
├────────────────────────────────────────────────────────────────────────┤
│ "A Watch event will be sent to the client EXACTLY ONCE when the        │
│  watched znode is mutated, after which the Watch is PURGED from memory!"│
└────────────────────────────────────────────────────────────────────────┘

1.1 The Race Condition Hazard¶

Timeline:
t0: Client reads data with watch: getData("/config", watch=true) ===> Returns "V1"
t1: Server updates: setData("/config", "V2") ===> Triggers & Purges Watch!
t2: Server sends WatchedEvent to client over TCP.
t3: [CRITICAL WINDOW: Watch is INACTIVE!]
    Another client issues: setData("/config", "V3")
t4: Client receives event and issues new watch: getData("/config", watch=true)

How ZooKeeper Solves the Gap:
At step t4, the client's getData("/config", watch=true) returns "V3" while atomically re-arming the watch. The client does not miss the state because the read returns the latest value as of the instant the new watch is installed!

2. Cellular Anatomy of `WatchManager`¶

ZooKeeper maintains watches in-memory using Bidirectional HashMaps (WatchManager.java):

public class WatchManager {
    // Maps a znode path to all client watchers listening on that path
    private final Map<String, Set<Watcher>> watchTable = 
        new HashMap<String, Set<Watcher>>();

    // Reverse mapping: Client connection -> Set of all paths it is watching
    private final Map<Watcher, Set<String>> watch2Paths = 
        new HashMap<Watcher, Set<String>>();
}

Bidirectional Memory Structures:
Path: "/services/db" ──► [ Client C1, Client C2, Client C5 ] (watchTable)
                             ▲
                             │ (Reverse Map for O(1) Cleanup on Disconnect!)
Client C1            ──► [ "/services/db", "/app/lock", "/leader" ] (watch2Paths)

2.1 The Disconnect Cleanup Superpower¶

When a client abruptly disconnects, ZooKeeper does not iterate through millions of znodes to remove its watches. It accesses watch2Paths.get(clientCnxn) and prunes only the specific registered paths in \(O(\text{Client Watches})\) time, preventing memory leaks in the JVM.

3. Event Types & In-Band Wire Dispatching¶

When a mutating transaction commits in FinalRequestProcessor, ZooKeeper evaluates matching watches and fires a WatchedEvent:

Event Constant	Triggering Action
`NodeCreated`	Target znode path was created via `create()`.
`NodeDeleted`	Target znode path was erased via `delete()`.
`NodeDataChanged`	Payload of target znode was modified via `setData()`.
`NodeChildrenChanged`	A child znode was created or deleted under this path.

3.1 Network Event Serialization¶

Crucially, a WatchedEvent contains NO data payloads: - It carries only: path, eventType, and keeperState. - This ensures event notifications are microscopic packets (a few tens of bytes), minimizing network interface card saturation.

4. Modern Persistent Recursive Watches (ZooKeeper 3.6+)¶

In legacy ZooKeeper, monitoring an entire subtree (e.g. /services/...) required registering individual one-time watches on every single child and grandchild znode, creating intense network churn.

ZooKeeper 3.6 introduced Persistent Recursive Watches:

// Modern Curator / ZooKeeper 3.6+ Persistent Watch
client.addWatch()
      .withMode(AddWatchMode.PERSISTENT_RECURSIVE)
      .forPath("/services");

- Persistent: Does not purge itself after firing; continues listening for all future events. - Recursive: Automatically covers all descendants, children, and nested subtrees under /services with a single server-side registration!