Phase 9: Real-World Choke Points, Eviction Tuning & Production Diagnostics¶

Audience: Principal Distributed Systems Architect & Production Operations Engineer
Source References: src/evict.c, src/lazyfree.c, src/latency.c, src/slowlog.c

1. The BigKey & HotKey Pathology¶

In a single-threaded engine, any operation that takes milliseconds directly blocks all other concurrent requests. The two most common causes of multi-second outages are BigKeys and HotKeys.

┌────────────────────────────────────────────────────────┐
│                   The BigKey Freeze                    │
└───────────────────────────┬────────────────────────────┘
                            │
                            ▼
              [ Client issues: DEL massive_hash ]
              (Hash contains 2,000,000 fields)
                            │
                            ▼
        [ Main Thread Trapped in Synchronous Free ]
        - Iterates through 2,000,000 dictEntry nodes
        - Invokes zfree() on 4,000,000 SDS pointers
        - CPU 100% saturated for 850 milliseconds!
                            │
                            ▼
    [ Catastrophic Ripple Effect: Health Check Timeout ]
    - Sentinel marks Master SDOWN / ODOWN
    - Premature failover triggered while Master was merely freeing memory!

1.1 Asynchronous Deletion (`UNLINK` and LazyFree)¶

To eliminate deletion freezes, Redis 4.0+ introduced asynchronous memory reclamation: - DEL: Synchronous. Frees all memory immediately on the main thread. - UNLINK: Asynchronous. Removes the key from the dictionary in $O(1)$ time, then delegates the heavy freeing of child nodes to the background BIO_LAZY_FREE thread.

Production Configuration:¶

lazyfree-lazy-eviction yes
lazyfree-lazy-expire yes
lazyfree-lazy-server-del yes
replica-lazy-flush yes

1.2 Detecting BigKeys & HotKeys in Production¶

# Scan keyspace non-blockingly to detect the largest data structures
redis-cli -p 6379 --bigkeys

# Track access frequency to pinpoint hot keys (Requires LFU policy or active tracking)
redis-cli -p 6379 --hotkeys

2. Eviction Policy Mechanics: Approximated LRU vs. LFU¶

When used_memory reaches maxmemory, Redis must evict keys according to maxmemory-policy.

2.1 Why Redis Does NOT Use a True LRU Linked List¶

A textbook LRU cache (like Python's functools.lru_cache) maintains a doubly linked list of all keys. Every read operation (GET) moves the accessed node to the head of the list: - Requires 2 pointers per key (16 bytes). - Creates intense CPU cache invalidation and pointer mutations on read-only queries. - For 100 million keys, the LRU list alone would consume 1.6 GB of RAM purely for pointers!

2.2 Redis Approximated LRU (`maxmemory-samples`)¶

Instead of a linked list, Redis uses Random Sampling: 1. Redis stores a 24-bit timestamp in robj->lru. 2. On every access, robj->lru is updated with the current clock. 3. During eviction, Redis randomly samples $N$ keys (controlled by maxmemory-samples, default: 5 or 10) into an eviction pool sorted by idle time. 4. The key with the highest idle time is evicted.

LRU Approximation Accuracy:
Samples = 5:  Approximates true LRU with ~80% accuracy.
Samples = 10: Mathematically indistinguishable from true LRU, with ZERO pointer overhead!

Random Keys Sampled (Samples = 5):
Key A (Idle: 120s) ──┐
Key B (Idle: 4s)   ──┼──► Eviction Pool ──► [Key C (Idle: 850s)] ──► EVICTED!
Key C (Idle: 850s) ──┘

2.3 Approximated LFU (Least Frequently Used) Internals¶

LFU tracks how often a key is accessed rather than merely when it was last touched. The 24-bit robj->lru field is split into two components:

┌───────────────────────────────────────┬───────────────────────────────┐
│ Last Decrement Time (LDT: 16 Bits)    │ Logistic Counter (LOG_C: 8B)  │
│ Resolution: 1 minute (0 to 65,535 min)│ Logarithmic rank (0 to 255)   │
└───────────────────────────────────────┴───────────────────────────────┘

How the 8-bit Logarithmic Counter Works:¶

An 8-bit counter can only count up to 255. To represent millions of accesses, Redis increments LOG_C probabalistically: $$P(\text{increment}) = \frac{1}{\text{counter} \times \text{lfu-log-factor} + 1}$$

New keys start with LOG_C = 5 so they are not immediately evicted.
As keys sit idle, serverCron decrements LOG_C based on lfu-decay-time (decay factor in minutes).

3. Latency Diagnostics & Slowlog Auditing¶

3.1 The Redis Slowlog (`SLOWLOG GET`)¶

The slowlog records commands whose execution time exceeded slowlog-log-slower-than (microseconds). Crucial: The slowlog measures strictly the execution time inside processCommand(), excluding network transfer and client queue wait times.

# View the 10 slowest commands
SLOWLOG GET 10

3.2 Automated Latency Engine (`LATENCY DOCTOR`)¶

CONFIG SET latency-monitor-threshold 10 # Track any internal event > 10ms
LATENCY LATEST
LATENCY DOCTOR

LATENCY DOCTOR analyzes the internal event stream and outputs human-readable diagnostic recommendations: - Detects high fork latency during BGSAVE. - Identifies appendfsync disk blocks. - Identifies slow clients saturating output buffers.