Phase 6: Distributed Primitives & Apache Curator Recipes

Audience: Principal Distributed Systems Architect & Application Platform Engineer
Source References: curator-recipes/src/main/java/org/apache/curator/framework/recipes/locks/

---

1. The Distributed Lock: Naive Stampede vs. Herd-Free Fair Lock

Implementing a distributed lock over a raw consensus engine is fraught with severe network stampede traps if designed incorrectly.

1.1 The Naive Lock Anti-Pattern (The Herd Effect Catastrophe)

• 1,000 workers want a lock.
• All 1,000 workers attempt: create("/exclusive_lock", EPHEMERAL).
• Worker 1 succeeds; remaining 999 workers fail and place a Watch on /exclusive_lock.
• When Worker 1 releases the lock:
• ZooKeeper fires 999 concurrent watch notifications simultaneously!
• All 999 workers wake up and bombard ZooKeeper with 999 create() requests at the exact same millisecond.
• The server CPU and network NIC saturate, dropping connections.

---

1.2 The Herd-Free Fair Lock Algorithm (Curator InterProcessMutex)

Apache Curator solves this by ordering requests into a sequential queue using Ephemeral Sequential Znodes:

Herd-Free Lock Queue under /locks/mutex/:
┌─────────────────────────────────┐
│ node-0000000001 (Worker A)      │ <=== OWNS THE LOCK!
└─────────────────────────────────┘
                 ▲ (Watched by)
┌─────────────────────────────────┐
│ node-0000000002 (Worker B)      │
└─────────────────────────────────┘
                 ▲ (Watched by)
┌─────────────────────────────────┐
│ node-0000000003 (Worker C)      │
└─────────────────────────────────┘

Curator InterProcessMutex Algorithm:
1. Client creates ephemeral sequential node:
   path = zk.create("/locks/mutex/lock-", EPHEMERAL_SEQUENTIAL)
   (e.g., /locks/mutex/lock-0000000002)

2. Client calls: children = zk.getChildren("/locks/mutex", watch=false)
3. Sorts children lexicographically by sequence number.

4. Evaluates Ownership:
   - Is my path the LOWEST sequence number in the list?
     - YES: Lock is granted! Proceed to critical section.
     - NO: Find the node IMMEDIATELY PRECEDING my sequence number:
       (e.g., node-0000000002 watches node-0000000001).

5. Places a Watch on ONLY the preceding node:
   zk.exists(preceding_path, watch=true)

6. When Worker A finishes and deletes node-0000000001:
   - ZooKeeper sends a Watch notification to EXACTLY ONE CLIENT (Worker B)!
   - Worker C remains completely asleep!
   - Network traffic is strictly O(1)! Zero Herd Effect!

---

2. Distributed Leader Election Recipes

Curator provides two battle-tested recipes for electing a leader across multiple worker microservices:

┌────────────────────────────────────────────────────────────────────────┐
│ 1. LeaderLatch                                                         │
├────────────────────────────────────────────────────────────────────────┤
│ - Blocks client thread until leadership is acquired: latch.await()    │
│ - Ideal for applications that boot up as passive standbys              │
│ - If leader crashes, the next latch in line unblocks automatically     │
└────────────────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────────────────┐
│ 2. LeaderSelector                                                      │
├────────────────────────────────────────────────────────────────────────┤
│ - Event-driven listener model: takeLeadership(CuratorFramework client) │
│ - Relinquishes leadership when method exits, allowing round-robin      │
│   leadership handoffs among cluster workers                            │
└────────────────────────────────────────────────────────────────────────┘

2.1 Production Java Implementation

CuratorFramework client = CuratorFrameworkFactory.newClient(
    "127.0.0.1:2181", 
    new ExponentialBackoffRetry(1000, 3)
);
client.start();

LeaderSelector selector = new LeaderSelector(client, "/election/master", 
    new LeaderSelectorListenerAdapter() {
        @Override
        public void takeLeadership(CuratorFramework client) throws Exception {
            System.out.println("I am now the authoritative master coordinator!");
            // Run master duties until interrupted or process shuts down...
            while (!Thread.currentThread().isInterrupted()) {
                Thread.sleep(1000);
            }
        }
    }
);
selector.autoRequeue();
selector.start();

---

3. Distributed Double Barriers (DistributedDoubleBarrier)

A Double Barrier coordinates distributed workers so that: 1. No worker begins execution until all \(N\) workers have arrived at the start. 2. No worker exits until all \(N\) workers have completed execution.

Double Barrier Lifecycle:
Phase 1: Enter Barrier
- Workers create ephemeral node under /barrier/nodes/worker-X.
- Watch child count: When count == N, all workers enter computing phase together.

Phase 2: Leave Barrier
- Upon finishing computation, workers delete their ephemeral node.
- When count == 0, all workers terminate cleanly.