Phase 5: Persistence Engines & Crash Recovery¶

Audience: Principal Database Reliability Engineer & Kernel Architect
Source References: src/rdb.c, src/aof.c, src/rio.c, src/aof_manifest.c

1. RDB Snapshotting: The `fork()` and Copy-on-Write (COW) Mechanics¶

Redis provides point-in-time point-in-time snapshots of the dataset via RDB (Redis Database) binary files.

┌────────────────────────────────────────────────────────┐
│                   BGSAVE Invocation                    │
└───────────────────────────┬────────────────────────────┘
                            │
                            ▼
              [ main thread calls fork() ]
              - Duplicates Page Table Entries (PTEs)
              - Marks all RAM pages as READ-ONLY
                            │
            ┌───────────────┴───────────────┐
            ▼                               ▼
    [Parent Process]                 [Child Process]
    Continues serving clients.       Iterates through redisDb.
    Writes trigger Page Fault:       Writes compact RDB binary file.
    OS copies dirty 4KB page.        Exits when complete.

1.1 The Fork Latency Spike¶

While fork() does not copy physical RAM immediately, it must copy the page table from parent to child. - For a 32 GB Redis instance using standard 4 KB pages, the page table contains \(8,388,608\) entries, consuming \(\approx 64 \text{ MB}\). - Copying a 64 MB page table locks the parent process CPU for 10 to 50 milliseconds. During this window, all client commands are completely blocked!

1.2 Copy-on-Write (COW) Memory Amplification¶

Once the child process begins writing the RDB file: 1. Every physical page in RAM is marked read-only by the kernel Memory Management Unit (MMU). 2. When a client issues a mutating command (e.g., SET key val), the CPU triggers a hardware Page Fault Exception. 3. The Linux kernel traps into do_wp_page(), allocates a new 4 KB physical memory frame, copies the 4 KB page content, updates the parent's page table, and marks the page writable.

The Mathematical Memory Hazard:¶

If your instance holds 20 GB of RAM and receives heavy write mutation during BGSAVE, the parent process may allocate an additional 15 GB of COW pages! If total memory exceeds physical RAM, the Linux Out-Of-Memory (OOM) killer sends SIGKILL to redis-server.

1.3 The Transparent Huge Pages (THP) Catastrophe¶

By default, Linux modern distributions enable Transparent Huge Pages (transparent_hugepage=always), which bundles memory into 2 MB pages instead of 4 KB pages. - Under COW, mutating a single 50-byte key forces the kernel to copy an entire 2 MB memory chunk instead of 4 KB. - Impact: - Memory consumption explodes by 500x. - Page fault latency spikes from 2 microseconds to 20 milliseconds. - Mandatory Kernel Fix:

echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

2. Append-Only File (AOF): Logging & `fsync` Policies¶

Unlike RDB snapshots (which lose data since the last snapshot), AOF logs every write command in RESP format:

[Client Command] ──► [Executed by Engine] ──► [Appended to server.aof_buf in RAM]
                                                              │
                                                              ▼ (beforeSleep)
                                               [write(fd, aof_buf) to OS Page Cache]
                                                              │
                                                              ▼ (appendfsync policy)
                                               [fsync(fd) to Physical NVMe/SSD]

2.1 The Three `appendfsync` Policies Compared¶

Policy	Durability Guarantee	Write Throughput	I/O Stalling Risk
`appendfsync always`	Zero loss (atomic fsync per command)	Very Low (\(\approx 2,000 \text{ ops/sec}\))	Extreme (Every command waits for disk write head)
`appendfsync everysec`	Maximum 1 to 2 seconds loss	Very High (\(\approx 100,000+ \text{ ops/sec}\))	Low (Handled by background BIO thread)
`appendfsync no`	OS determines flush (typically 30s)	Maximum (Network/RAM bound)	Zero inside Redis; high unpredictability

2.2 The Background `fsync` Stall Trap¶

In appendfsync everysec, a background thread (src/bio.c) issues fsync(). However: 1. If the disk subsystem is saturated by heavy I/O, fsync() blocks in the BIO thread. 2. The main thread checks if a background fsync has been pending for more than 2 seconds. 3. If it has, the main thread refuses to call write() and blocks execution, causing client latency spikes to jump from 100 microseconds to several seconds!

Defensive Setting:¶

no-appendfsync-on-rewrite yes

(Instructs Redis not to call fsync while a BGSAVE or AOF rewrite is active, preventing disk saturation stalls at the cost of durability during rewrites).

3. Multi-Part AOF (MP-AOF) Revolution in Redis 7.0+¶

Prior to Redis 7.0, AOF rewriting suffered severe issues: 1. The parent had to accumulate mutating commands in an AOF Rewrite Buffer in RAM. 2. Large write volumes during rewrite caused multi-gigabyte RAM spikes and final-stage flush freezes.

3.1 MP-AOF Architecture¶

In Redis 7.0, AOF is no longer a monolithic file. It is a directory containing three file roles:

/var/lib/redis/appendonlydir/
├── appendonly.aof.manifest            <--- Atomic state ledger
├── appendonly.aof.1.base.rdb          <--- RDB snapshot base file
├── appendonly.aof.1.incr.aof          <--- Incremental log before rewrite
└── appendonly.aof.2.incr.aof          <--- New active incremental log

Multi-Part AOF Lifecycle:
1. Normal Operation:
   - Base file (historical snapshot) + Active Incremental file (new writes).

2. Rewrite Triggered (BGREWRITEAOF):
   - Redis opens a NEW Incremental file (e.g., incr.2.aof) for incoming writes.
   - Child process forks and writes a brand new Base file (base.2.rdb).
   - Zero in-memory rewrite buffers needed!

3. Rewrite Finishes:
   - Child exits cleanly.
   - Parent updates appendonly.aof.manifest via atomic rename().
   - Background BIO thread deletes old base.1.rdb and incr.1.aof asynchronously.

3.2 Structure of the Manifest File (`appendonly.aof.manifest`)¶

file appendonly.aof.2.base.rdb seq 2 type b
file appendonly.aof.2.incr.aof seq 2 type i

4. Hybrid Persistence & Crash Recovery¶

For the fastest recovery times with minimal data loss, production systems configure Hybrid Persistence:

aof-use-rdb-preamble yes

4.1 How Hybrid Persistence Works¶

When the base AOF file is created, Redis writes it using the RDB binary format rather than text-based RESP commands. Subsequent incremental files are written in pure RESP text.

On Server Crash Recovery:
1. Fast Binary Phase:
   - Redis loads the binary RDB base file into RAM using direct memory deserialization.
   - Speed: ~1 GB/second (orders of magnitude faster than parsing text RESP commands).
2. Delta Replay Phase:
   - Redis replays the small incremental RESP commands from incr.aof to restore recent state.

Result: A 50 GB dataset reboots and achieves operational readiness in seconds instead of taking hours to parse raw text commands.