Skip to content

Cache Avalanche

1. What is it? (2-3 lines)

A Cache Avalanche occurs when a large volume of cached keys expire simultaneously at the exact same timestamp (or when a central Redis/Memcached cluster crashes), causing a massive wave of read traffic to hit the downstream primary database at once, triggering a total database outage.

2. Why does it exist?

Developers frequently set identical fixed TTLs (e.g. EXPIRE key 3600) across millions of items during batch writes or cache warming. When the fixed 1-hour window elapses, millions of keys expire in the exact same second. The resulting sudden 100x traffic surge to the primary database causes CPU/RAM overload and database collapse.

3. Where is it used?

  • Mass Cache Key Expiration: Batch jobs setting uniform TTLs on product catalogs, category trees, or user session data.
  • Cache Cluster Failover: Sudden crash or reboot of a primary Redis cache node without multi-replica failover.

4. How does it work? (simple flow)

  1. Uniform TTL Setting: At 02:00 AM, batch job writes 500,000 product keys with TTL = 7200s (2 hours).
  2. Mass Simultaneous Expiration: At 04:00 AM, all 500,000 keys expire at the exact same millisecond.
  3. Traffic Flood: All subsequent incoming HTTP requests miss the cache simultaneously.
  4. Database Crash: Primary DB receives 50,000 QPS of complex SQL queries \(\implies\) CPU reaches 100% \(\implies\) Database connection pool exhausts \(\implies\) System outage.

5. Solutions & Mitigations

  • Randomized Jitter TTL: Add a random delta to fixed TTLs: TTL = Base_TTL + Random_Jitter (e.g., 3600s + rand(0, 600s)), spreading key expirations smoothly across a 10-minute window.
  • High-Availability (HA) Cache Clustering: Deploy Redis Sentinel / Redis Cluster with multi-AZ replication to prevent total cache cluster outages.
  • Circuit Breakers & Rate Limiting: Limit max concurrent database queries during cache misses to protect database health.
  • Multilevel Caching (L1/L2): Combine local in-memory L1 cache (Caffeine) with distributed L2 cache (Redis).

6. Advantages & Limitations of Solutions

  • Randomized Jitter: Simple 1-line code fix (TTL + rand()); 100% eliminates simultaneous TTL expiration avalanches with zero performance penalty.
  • HA Replication: Prevents infrastructure crashes from causing avalanches; increases Redis cloud infrastructure costs.

7. Common interview questions (10)

  1. How does a Cache Avalanche differ from a Cache Stampede and Cache Penetration?
  2. What is the mathematical formula for adding Randomized Jitter to cache TTLs?
  3. How does a Redis Cluster failover (Redis Sentinel) mitigate Cache Avalanche caused by node crashes?
  4. How do Multilevel Caches (L1 Guava/Caffeine + L2 Redis) protect databases during a Cache Avalanche?
  5. How can a Circuit Breaker prevent a Cache Avalanche from crashing the primary relational database?
  6. What is the role of Cache Warming after a Redis cluster reboot to prevent an Avalanche?
  7. How does rate limiting on cache miss paths protect database connection pools?
  8. What happens when an entire Redis node experiences an Out-Of-Memory (OOM) eviction sweep?
  9. How do CDN Edge caches protect backend API microservices from Cache Avalanches?
  10. How do you design a self-healing caching tier that degrades gracefully during an avalanche?

8. Common misconceptions

  • Misconception: Cache Avalanche only occurs when Redis crashes. Fact: Avalanche most commonly occurs on a fully healthy Redis server due to synchronized TTL expiration windows!
  • Misconception: Increasing database CPU cores prevents an Avalanche. Fact: A 100x sudden traffic surge will overwhelm a database regardless of reasonable CPU allocation.

10. 30-second cheat sheet

  • Problem: Millions of keys expire at the exact same second OR Redis cluster crashes \(\implies\) DB collapses.
  • Primary Fix: Randomized TTL Jitter \(\implies\) TTL = 3600 + rand(0, 600). Spreads expirations over time!
  • Infrastructure Fix: Deploy Redis Cluster / Sentinel with multi-AZ read replicas.
  • Protection: Use Circuit Breakers & Multilevel Caches (L1 In-Memory + L2 Redis).