Skip to content

Distributed Locks

1. What is it? (2-3 lines)

A Distributed Lock is a mutual exclusion mechanism across independent application nodes in a distributed cluster, ensuring that only one process can execute a critical section or access a shared resource at any given time.

2. Why does it exist?

Single-process locks (e.g. Java ReentrantLock or Go sync.Mutex) operate strictly within one process memory space. In a clustered microservice environment with 50 application nodes, a single-process lock cannot prevent multiple nodes from simultaneously running duplicate cron jobs, double-charging a user, or updating shared storage concurrently.

3. Where is it used?

  • In-Memory Caches: Redis (SET NX PX, Redlock algorithm).
  • Consensus / Coordination Stores: etcd (Leases), Apache ZooKeeper (Ephemeral Sequential Znodes), Consul.
  • Database Locks: PostgreSQL pg_advisory_lock.

4. How does it work? (simple flow)

  • Redis (SET NX PX) Flow:
  • Process A attempts acquire: SET lock:order:123 random_uuid NX PX 30000 (Sets key only if Not Exists with 30s TTL).
  • If success: Process A holds lock \(\to\) Executes critical section.
  • Release: Process A executes Atomic Lua Script verifying key_value == random_uuid before DEL (prevents deleting another process's lock!).
  • If process crashes: TTL expires automatically, preventing permanent deadlocks.
  • Fencing Tokens (Correctness Guarantee): Lock service returns a monotonically increasing integer (fencing_token = 83). Storage layer rejects writes if fencing_token < last_seen_token.

5. Advantages

  • Prevents double execution, duplicate billing, and race conditions across microservice clusters.
  • Low latency when implemented on in-memory stores like Redis.
  • Automatic TTL safety net prevents permanent deadlocks on node crashes.

6. Limitations

  • GC Pause / Clock Drift Race Condition: A long Stop-The-World (STW) GC pause can cause lock TTL to expire while Process A is paused; Process B acquires lock, causing concurrent execution!
  • Requires Fencing Tokens at storage layer for 100% mathematical safety (Martin Kleppmann's Redlock critique).

7. Common interview questions (10)

  1. How does a single-instance Redis distributed lock (SET NX PX) work?
  2. Why is an atomic Lua script mandatory when releasing a Redis distributed lock?
  3. What is Martin Kleppmann's critique of the Redis Redlock algorithm regarding Clock Drift and GC pauses?
  4. How do Fencing Tokens protect against stale writes when a process suffers a STW GC pause?
  5. How does ZooKeeper implement Distributed Locks using Ephemeral Sequential Znodes and Watchers?
  6. How does etcd implement Distributed Locks using Keep-Alive Leases and Revision numbers?
  7. What is the Thundering Herd Problem in distributed locking and how do ZooKeeper Watchers eliminate it?
  8. When should an engineer use an Atomic Database Operation (UPDATE ... WHERE) instead of a Distributed Lock?
  9. How does Redisson Watchdog implement automatic lock TTL extension in Java Redis clients?
  10. What is the difference between Performance Locks (Redis) and Safety-Critical Locks (etcd/ZooKeeper)?

8. Common misconceptions

  • Misconception: Releasing a Redis lock can be done via standard DEL key. Fact: DEL without checking token value will accidentally delete a lock acquired by another thread after your TTL expired!
  • Misconception: Distributed locks replace atomic SQL operations. Fact: Atomic SQL UPDATE account SET balance = balance - 10 WHERE balance >= 10 is far safer and faster than a distributed lock.

10. 30-second cheat sheet

  • Redis Acquire: SET key random_uuid NX PX 30000 (Atomic set + TTL).
  • Redis Release: Atomic Lua Script checking GET(key) == random_uuid before DEL.
  • ZooKeeper Approach: Ephemeral Sequential Znodes + Watcher on previous node (No Thundering Herd).
  • Fencing Token: Monotonic integer (token = 83) passed to storage to reject stale writes from paused nodes.