Distributed Locks¶

1. Decision¶

Implement a Distributed Lock (Redis SET NX PX / Redlock, ZooKeeper, etcd) when mutual exclusion across multiple independent application instances is required to prevent race conditions during non-atomic multi-step operations.

2. 30-Second Answer¶

A Distributed Lock guarantees that among multiple application nodes in a cluster, only one process can hold a named lock at any given time to execute a critical section. Standard single-instance locks (e.g. Java ReentrantLock or Go sync.Mutex) operate only within a single process memory space. Distributed Locks utilize centralized, highly available stores: In-Memory Caches (Redis) for ultra-fast, performance-focused locking via atomic SET resource_name token NX PX 30000 commands, or Consensus Stores (etcd, ZooKeeper) for strict correctness-focused locking via lease heartbeats and ephemeral z-nodes. Distributed locks must enforce explicit TTL lease expirations and random fencing tokens to protect against node crashes and network pauses.

3. Decision Matrix¶

Option	Best For	Avoid When	Latency	Scalability	Consistency	Operational Complexity
Redis Single-Instance (`SET NX PX`)	High-performance locking where occasional lock loss during master crash is acceptable	Financial safety-critical operations requiring strict zero-double-execution guarantees	Sub-millisecond (<1ms)	Extremely High (Microsecond in-memory locks)	Weak during master node failover	Low
Redis Redlock Algorithm	Distributed locking across 5 independent Redis instances for improved fault tolerance	Ultra-high QPS locking, clock-drift sensitive financial platforms	Low (1-5ms across 5 Redis nodes)	High	Medium to High (Subject to Martin Kleppmann's clock drift critique)	Medium
Consensus Store (etcd / ZooKeeper)	Safety-critical distributed locking, leader election, infrastructure orchestration	High-velocity microsecond locking (>50,000 locks/sec)	Medium (5-20ms disk WAL + Raft/Paxos consensus)	Medium	Strong (Linearizable lease consensus)	High (Requires managing etcd/ZooKeeper cluster)

4. Selection Rules¶

Requirement	Best Choice	Rationale / Example
Performance-Focused Short-Lived Locks	Redis `SET NX PX`	Prevent 10 background cron workers from running the exact same 5-second task
Correctness-Focused Safety-Critical Locks	etcd / ZooKeeper	Infrastructure Leader Election or strictly single-writer storage volume mounting
Atomic Lock Release Protection	Lua Script Verification	Delete lock key only if key payload matches the worker's unique random token
Protect Against Long GC Pause Race Conditions	Fencing Tokens	Pass monotonically increasing `fencing_token` to storage engine to reject stale writes
Prevent Permanent Deadlocks on Node Crash	Mandatory Lock TTL Expiration	Always set an explicit expiration lease timeout (e.g. `PX 30000` = 30 seconds)

5. Tradeoffs¶

In-Memory Speed (Redis) vs Consensus Safety (etcd)
Gain (Redis): Sub-millisecond lock acquisition and release.
Lose (Redis): Asynchronous replication means primary node crash can lose a newly granted lock.
Lock Expiration TTL Safety vs Long Execution Window
Gain: Prevents deadlocks if the holding node crashes mid-execution.
Lose: If execution outlasts TTL, lock expires prematurely while worker is still running!
Fencing Tokens Security vs Storage Engine Integration
Gain: Protects database writes against paused workers whose lock TTL expired.
Lose: Downstream database must support atomic fencing token verification on writes.
Redlock Multi-Node Security vs Clock Drift Sensitivity
Gain: Acquires locks across 5 independent Redis nodes; tolerates 2 node failures.
Lose: Relies on system clock drift assumptions; criticized by distributed systems researchers (Kleppmann).
Distributed Lock Overhead vs Atomic Database Operations
Gain: Enables mutual exclusion across heterogeneous microservice code blocks.
Lose: Network overhead acquiring/releasing locks; database UPDATE ... WHERE atomic calls are often better.

6. Interview Counter Questions¶

How do you implement a single-instance Redis distributed lock using SET resource_name token NX PX 30000?
Why is a Lua script mandatory when releasing a Redis distributed lock?
What is Martin Kleppmann's critique of the Redis Redlock algorithm (Clock Drift vs GC Pauses)?
How do Fencing Tokens prevent stale writes when a process experiences a Stop-The-World (STW) GC pause?
How does ZooKeeper implement Distributed Locks using Ephemeral Sequential Nodes?
How does etcd handle lease renewal (Heartbeat Keep-Alive) for active distributed locks?
What happens if an application worker holding a distributed lock executes for 40s when lock TTL is 30s?
When should you use an Atomic Database Operation (UPDATE account SET balance = balance - 100 WHERE balance >= 100) instead of a Distributed Lock?
What is the Thundering Herd Problem in distributed lock acquisition and how do Watchers solve it in ZooKeeper?
How do you choose the right Lock TTL duration?

7. Gold Standard Answers¶

Acquire: SET lock:order:123 random_uuid NX PX 30000. NX ensures key is set only if it doesn't exist; PX 30000 sets a 30-second TTL expiration.
Releasing requires checking if the lock still belongs to this worker before deleting. A Lua script executes if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) else return 0 end atomically in RAM.
Kleppmann proved Redlock fails safety if system clocks drift or if a process suffers a STW GC pause after acquiring the lock but before writing to storage, causing 2 workers to hold the lock simultaneously.
The lock service issues a monotonically increasing integer (fencing_token = 83). The storage engine accepts writes only if fencing_token > last_seen_token. A paused worker holding token 83 has its write rejected if token 84 was already written.
Client creates an Ephemeral Sequential znode /locks/lock-000001. Client checks if its znode has the lowest sequence number. If yes, lock acquired. If no, client sets a Watcher on the immediately preceding znode.
etcd issues a Lease with a Time-To-Live (e.g., 10s). The client application runs a background Keep-Alive thread that periodically sends heartbeats to renew the lease until work completes.
The lock expires automatically at 30s. Another worker acquires the lock. Worker 1 finishes at 40s and corrupts shared state unless Fencing Tokens or a Lock Extension Thread (Redisson Auto-Renewal) are used.
Use Atomic Database Operations whenever possible! Single SQL atomic statements execute at storage engine level without network lock coordination, eliminating distributed lock complexity entirely.
Thundering Herd occurs when 1,000 workers poll Redis constantly for a released lock. ZooKeeper Watchers allow 999 workers to sleep until ZooKeeper notifies only the next in line upon znode deletion.
Lock TTL = Expected Max Execution Time \(\times 3\). Combine with a background lock extension thread (e.g., Redisson watchdog) that renews TTL every 10 seconds while execution continues.

8. Common Mistakes¶

Releasing a Redis lock via standard DEL key without verifying the random token via Lua script, deleting another thread's lock!
Using a plain Redis distributed lock for critical financial ledgers without Fencing Tokens or atomic DB operations.
Setting a static 5-second lock TTL for tasks that take 10 seconds, causing double execution race conditions.
Polling Redis in a tight while(true) loop to acquire a lock, burning CPU and Redis QPS.
Using Distributed Locks when a simple atomic SQL UPDATE statement or database UNIQUE constraint would cleanly solve the problem.

9. Keywords¶

Distributed Lock, Redis, SET NX PX, Lua Script, Redlock, etcd, ZooKeeper, Ephemeral Sequential Node, Fencing Token, Martin Kleppmann, STW GC Pause, Watchdog, Keep-Alive Lease, Atomic SQL.

10. Cheat Sheet¶

Single Redis Lock: Acquire via SET key random_token NX PX 30000. Release via Lua Script checking random_token.
Fencing Tokens: Always pass monotonically increasing fencing_token to storage engines to reject stale writes from paused workers.
Lock Safety Spectrum: High-Speed Caching Locks \(\rightarrow\) Redis SET NX; High-Safety Leaders/Locks \(\rightarrow\) etcd / ZooKeeper.
Auto-Renewal: Use a background watchdog thread to extend lock TTL while execution is actively running.
ZooKeeper Advantage: Uses Ephemeral Sequential Znodes + Watchers to notify next worker in line without tight polling loops.
Golden Rule: Avoid Distributed Locks if possible! Prefer Atomic SQL statements (UPDATE ... WHERE) or DB UNIQUE constraints.