Hot Keys¶

1. Decision¶

Mitigate Hot Key bottlenecks (single cache keys receiving extreme QPS that saturate single Redis shard CPU/network resources) by implementing Local In-Memory Caching (L1), Key Salting / Replication across shards, and Read-Replication scaling.

2. 30-Second Answer¶

A Hot Key issue occurs when a single cache key receives disproportionately high traffic volume (e.g., 100,000+ QPS for a celebrity profile, breaking news article, or flash sale item). In distributed caches like Redis Cluster, keys map to specific Hash Slots assigned to a single physical shard node. Excessive QPS targeting a single hot key saturates that specific node's CPU single-threaded event loop and network bandwidth, triggering latency spikes and dropped requests while sibling shard nodes sit idle. Mitigate Hot Keys by placing an L1 Local Cache in application memory, Salting/Replicating keys across multiple Redis shards (key_1, key_2, key_N), or scaling read-replicas.

3. Decision Matrix¶

Option	Best For	Avoid When	Latency	Scalability	Consistency	Operational Complexity
L1 Local In-App Cache (Caffeine)	Extreme hot key traffic (>100k QPS), static read-heavy hot objects	Rapidly changing volatile state requiring instant global synchronization	Ultra-Low (<0.1ms direct memory lookup)	Absorbs 99% of hot key traffic before Redis	Weak (Brief local staleness window per app node)	Low
Key Salting / Shard Replication	High-velocity hot key reads where local cache memory is insufficient	Dynamic write-heavy keys requiring atomic updates	Low (1ms Redis lookup)	Spreads hot key reads linearly across N Redis shards	Weak (Must write to N salted keys on update)	Medium
Redis Read Replicas + Load Balancing	Moderate hot key reads across cluster, read-heavy workloads	High write volume (Replication lag overhead)	Low (1ms Redis read)	Scales read capacity across replica nodes	Eventual (Subject to replica lag)	Medium

4. Selection Rules¶

Requirement	Best Choice	Rationale / Example
Extreme Hot Read Traffic (>100,000 QPS)	L1 Local Cache	App memory absorbs reads natively; zero Redis network hops
Distribute Hot Key Load Across Redis Cluster	Key Salting	Append random suffix `hotkey:#{rand(1,N)}` to scatter reads across N shards
Detect Hot Keys Dynamically in Real-Time	Redis `--hotkeys` / LFU Sampling	Identify hot keys using Redis memory sampling or proxy traffic metrics
Hot Key Invalidation Across Salting Shards	Multi-Key Update / Pub-Sub	Broadcast updates to all salted key replicas (`hotkey:1` through `hotkey:N`)
Prevent Cache Breakdown on Hot Key Expiration	Mutex Lock / Probabilistic Refresh	Ensure only 1 worker refreshes DB when hot key expires (XFetch)

5. Tradeoffs¶

L1 Local Caching vs Centralized Redis Routing
Gain (L1): Completely shields Redis hardware infrastructure from hot key traffic.
Lose (L1): Introduces cache drift across application nodes; duplicates hot key memory usage in app JVM heap.
Key Salting (Shard Scatter) vs Single Key Integrity
Gain (Salting): Linearly scales hot key read throughput across \(N\) physical Redis shards.
Lose (Salting): Writes must update all \(N\) salted keys (hotkey_1 to hotkey_N), creating write amplification.
Read Replica Scaling vs Replication Lag
Gain: Increases read throughput capacity by routing queries to secondary replicas.
Lose: Replicas may serve stale data during replication lag windows under heavy load.
Dynamic Hot Key Detection vs Static Pre-Salting
Gain (Dynamic): Automatically detects and caches emerging trending keys without manual configuration.
Lose (Dynamic): Requires traffic proxy monitoring overhead (e.g., Envoy / Redis LFU inspection).

6. Interview Counter Questions¶

Why does Redis Cluster fail to balance traffic for a single Hot Key across shards automatically?
How do you implement Key Salting (Random Suffixing) to distribute hot key reads across Redis shards?
What is Write Amplification in Key Salting and how do you minimize it?
How do you detect Hot Keys in real-time without impacting production Redis performance?
How does a Two-Tier (L1 Local + L2 Distributed) caching model solve the Hot Key problem?
What is the difference between a Hot Key problem and a Big Key problem in Redis?
How does Envoy proxy or Twemproxy help detect and handle Hot Keys at the infrastructure layer?
What happens when a Hot Key expires (Hot Key Cache Breakdown) and how do Mutex locks prevent DB collapse?
How do you invalidate all salted key instances (hotkey:1 through hotkey:10) consistently during writes?
Which system design scenarios commonly trigger severe Hot Key bottlenecks (e.g., Flash Sales, Twitter/X)?

7. Gold Standard Answers¶

Redis Cluster partitions key space using Hash Slots (CRC16(key) mod 16384). All requests for a single exact key string map to one specific Hash Slot on a single physical shard node, making automatic shard load balancing impossible.
During write: Write payload to \(N\) salted keys (key:1, key:2, ..., key:N) across shards. During read: Application picks a random index key:rand(1, N) to query, distributing traffic evenly across \(N\) shards.
Write Amplification occurs because updating 1 value requires \(N\) network write calls to update all \(N\) salted key copies. Minimize by pairing Key Salting with short TTLs and updating via async background workers or Redis Pub/Sub.
Use Redis LFU eviction policy and run redis-cli --hotkeys (uses low-overhead sampling). Alternatively, inspect metrics at the API Gateway or Redis Proxy layer (Envoy/Twemproxy) to track request frequency per key.
L1 Local Cache (Caffeine in app memory) intercepts reads. The first request fetches the hot key from Redis into local RAM; subsequent 100,000 reads hit local RAM (\(<0.1\text{ms}\)), completely bypassing Redis.
Hot Key = Single key receiving extreme QPS (saturates CPU/network bandwidth). Big Key = Single key holding massive payload size (e.g., 500MB Hash/Set, causes memory allocation and serialization spikes).
Envoy proxy tracks key access frequency in memory filters. When a key exceeds a QPS threshold, Envoy automatically caches the response at the proxy layer for a few seconds, shielding Redis backend servers.
When a hot key expires, thousands of concurrent requests miss cache simultaneously (Cache Breakdown). Use a Distributed Mutex Lock: the first thread acquires the lock to query DB and update cache; other threads wait or return stale data.
Write to primary DB, then publish an invalidation event over Redis Pub/Sub. Worker processes consume the event and execute DEL key:1 key:2 ... key:N or issue async deletions across all shards.
Flash sales (iPhone launch inventory counter), celebrity tweets (viral post comments), breaking news homepage articles, and global live sports streaming leaderboard scores.

8. Common Mistakes¶

Assuming adding more Redis Cluster shards will automatically solve a single Hot Key bottleneck (1 key = 1 shard!).
Running un-sampled redis-cli --hotkeys or heavy monitoring scripts during peak production traffic hours.
Forgetting Write Amplification when applying Key Salting, causing heavy write performance degradation.
Allowing Hot Keys to expire without Mutex locks or Probabilistic Early Expiration, causing DB collapse.
Confusing Hot Keys (high QPS) with Big Keys (large payload size) and applying the wrong remediation strategy.

9. Keywords¶

Hot Key, Hash Slot, CRC16, Key Salting, Write Amplification, L1 Local Cache, Caffeine, Read Replicas, Mutex Lock, Cache Breakdown, Big Key, Envoy Proxy, Redis LFU, --hotkeys.

10. Cheat Sheet¶

Core Rule: Single Key = Single Hash Slot = Single Redis Shard. Adding cluster nodes does NOT scale a single hot key!
Best Fix: L1 Local Cache (Caffeine) in application memory. Absorbs 99% of hot key reads directly in RAM.
Alternative Fix: Key Salting: Write to key:1 .. key:N; read randomly from key:rand(1, N) to split load across N shards.
Write Penalty: Key Salting creates Write Amplification (\(N\) writes for 1 key update).
Expiration Defense: Use Mutex locks or XFetch probabilistic refresh to prevent DB collapse when a hot key expires.
Hot vs Big Key: Hot Key = High QPS on single key; Big Key = Large byte size of single key payload.