Skip to content

TheOpenPrep

Redis Master Memory Map: The Connected Neural Architecture

Redis Master Memory Map: The Connected Neural Architecture¶

This document is your single-glance visual memory map and revision blueprint for Redis. Every component, C struct, memory arena, background thread, persistence file, and cluster socket is mapped as a connected graph so you can trace how every dot connects to every other dot in the Redis engine.

1. The Master Visual Memory Maps (Modular Deep-Zoom System)¶

All diagrams follow a uniform top-to-bottom (graph TD) layout with consistent color-coded nodes.

1.1 The Executive Macro Ecosystem (The 30,000-Foot View)¶

graph TD
    classDef client fill:#e1f5fe,stroke:#0288d1,color:#01579b;
    classDef engine fill:#ede7f6,stroke:#512da8,color:#311b92;
    classDef mem fill:#e8f5e9,stroke:#388e3c,color:#1b5e20;
    classDef disk fill:#fbe9e7,stroke:#d84315,color:#bf360c;
    classDef net fill:#e0f2f1,stroke:#00796b,color:#004d40;

    subgraph CLIENT_TIER ["1. Client & Network Ingress Tier"]
        C1[Client Applications]:::client -->|TCP Sockets| IOT["Threaded I/O Workers: Socket Read/Write"]:::net
        IOT -->|Parsed Commands| EL["ae Event Loop: Single-Threaded Core"]:::engine
    end

    subgraph MEMORY_TIER ["2. In-Memory Keyspace & Allocator Tier"]
        EL <-->|"Dict / SDS / Skiplist"| DB[("redisDb Keyspace")]:::mem
        DB <-->|"Arenas & Bins"| JEM[Jemalloc Memory Allocator]:::mem
        EL -->|Eviction Pool| EVIC["LRU / LFU Maxmemory Engine"]:::mem
    end

    subgraph PERSISTENCE_TIER ["3. Durability & Disk Persistence Tier"]
        EL -->|fork Copy-on-Write| RDB["RDB Snapshot: dump.rdb"]:::disk
        EL -->|Sequential Buffer| AOF["AOF Log: appendonly.aof"]:::disk
    end

    subgraph CLUSTER_TIER ["4. High Availability & Cluster Tier"]
        EL -->|Replication Stream| REPL[Replicas via PSYNC2]:::net
        EL <-->|Cluster Bus Port 16379| GOSSIP["Redis Cluster: 16384 Hash Slots"]:::net
    end

1.2 Regional Map 1: The Event Loop & Command Processing Engine¶

graph TD
    classDef proc fill:#ede7f6,stroke:#512da8,color:#311b92;
    classDef mem fill:#e8f5e9,stroke:#388e3c,color:#1b5e20;
    classDef priv fill:#fff3e0,stroke:#f57c00,color:#e65100;

    EPOLL["Kernel I/O Multiplexer: Epoll / Kqueue"]:::proc -->|Fires Socket Event| AE[aeProcessEvents Engine]:::proc
    AE -->|File Events| READ[readQueryFromClient]:::proc
    AE -->|Time Events| CRON["serverCron: 100ms Periodic Tasks"]:::proc

    READ -->|Appends raw bytes| QBUF["client->querybuf: SDS Buffer"]:::priv
    QBUF -->|Splits into Arguments| ARGV["client->argv: robj Array"]:::priv
    ARGV -->|Lookup Command Name| CMD["server.commands: Command Table Hash"]:::mem

    CMD -->|Atomic Execution| EXEC["proc Function Pointer: e.g. setCommand"]:::proc
    EXEC -->|Modifies Keyspace| DB[("server.db: dict")]:::mem
    EXEC -->|Appends Response| OUTBUF["client->buf: 16KB Static Buffer"]:::priv

    AE -->|beforeSleep Loop| FLUSH["writeToClient: Flushes outbuf to Socket"]:::proc
    OUTBUF --> FLUSH

1.3 Regional Map 2: Cellular Anatomy of Redis Data Structures¶

graph TD
    classDef obj fill:#ede7f6,stroke:#512da8,color:#311b92;
    classDef struct fill:#e8f5e9,stroke:#388e3c,color:#1b5e20;
    classDef mem fill:#fff3e0,stroke:#f57c00,color:#e65100;

    KSPC[redisDb dict]:::obj -->|Points to Value| ROBJ["redisObject: 16 Bytes"]:::obj

    ROBJ --- T["type: 4 bits String/List/Set/ZSet/Hash"]:::struct
    ROBJ --- E["encoding: 4 bits raw/embstr/listpack/skiplist"]:::struct
    ROBJ --- L["lru: 24 bits LRU timestamp or LFU counter"]:::struct
    ROBJ --- R["refcount: 32 bits shared object counter"]:::struct
    ROBJ --- P["ptr: 64-bit Pointer to Actual Payload"]:::struct

    P -->|If String| SDS["SDS: Simple Dynamic String"]:::mem
    SDS --- SDSH["sdshdr8: len, alloc, flags, buf"]:::mem

    P -->|"If Small Hash / List"| LP["Listpack: Contiguous Compact Array"]:::mem
    LP --- LPE[Total Bytes, Num Elements, Back-length entries]:::mem

    P -->|If Large List| QL["Quicklist: Doubly-Linked List of Listpacks"]:::mem

    P -->|If Sorted Set ZSET| ZS[zset Dual Structure]:::mem
    ZS -->|O 1 Score Lookup| ZD["dict: member to score"]:::struct
    ZS -->|O logN Range Search| ZSL["zskiplist: Forward pointers and Spans"]:::struct

1.4 Regional Map 3: Persistence, Replication & Clustering Pipeline¶

graph TD
    classDef proc fill:#ede7f6,stroke:#512da8,color:#311b92;
    classDef mem fill:#e8f5e9,stroke:#388e3c,color:#1b5e20;
    classDef disk fill:#fbe9e7,stroke:#d84315,color:#bf360c;
    classDef net fill:#e0f2f1,stroke:#00796b,color:#004d40;

    MAIN[Main Redis Process]:::proc -->|"BGSAVE: Linux fork"| CHILD["Child Process: RDB Dumper"]:::proc
    MAIN -.->|Modifies memory during dump| COW[Linux OS Copy-on-Write Page Split]:::mem
    CHILD -->|Serializes RAM to disk| RDBF[("dump.rdb Binary File")]:::disk

    MAIN -->|Every mutation| AOFBUF["server.aof_buf: SDS Memory Buffer"]:::mem
    AOFBUF -->|write system call| OSBUF[OS Page Cache Kernel Buffer]:::mem
    OSBUF -->|"fsync policy: everysec"| AOFF[("appendonly.aof Disk File")]:::disk

    MAIN -->|Feeds mutation stream| REPLBUF["server.repl_backlog: Circular Ring Buffer"]:::mem
    REPLBUF -->|PSYNC2 replication stream| SLAVE[Replica Node]:::net
    MAIN <-->|Cluster Bus Port 16379| GOSSIP["Gossip Mesh: 16384 Hash Slots"]:::net

2. The 3 Primary Pathways in Motion (Visual Interactive Traces)¶

Pathway A: Trace of a Write Command (`SET user:100 "active" EX 3600`)¶

sequenceDiagram
    autonumber
    actor Client as "Client App"
    participant Epoll as "Epoll Event Loop"
    participant BE as "Redis Main Thread"
    participant Dict as "redisDb (dict)"
    participant Expire as "db->expires (dict)"
    participant AOF as "aof_buf"
    participant Repl as "repl_backlog"

    Client->>Epoll: Sends TCP packet: *4\r\n$3\r\nSET...
    Epoll->>BE: Fires readable event on client socket
    BE->>BE: readQueryFromClient(): Parses arguments into client.argv
    BE->>BE: Lookup "set" in server.commands table
    BE->>Dict: dictAddOrUpdate(db.dict, "user:100", "active")
    BE->>Expire: dictAdd(db.expires, "user:100", now() + 3600s)
    BE->>AOF: feedAppendOnlyFile(): Appends raw RESP command
    BE->>Repl: replicationFeedSlaves(): Pushes into circular ring buffer
    BE->>Client: addReply(): Writes "+OK\r\n" to client.buf (16KB static)
    Note over BE,Client: beforeSleep() flushes client.buf to socket!

Pathway B: Trace of Memory Eviction & Expiration¶

sequenceDiagram
    autonumber
    participant Client as "Incoming Write Query"
    participant Server as "server.c Engine"
    participant Expire as "Active Expire Cycle"
    participant Evic as "Eviction Pool (Maxmemory)"
    participant JEM as "Jemalloc Allocator"

    Note over Expire: serverCron runs every 100ms:
    Expire->>Expire: activeExpireCycle(): Samples 20 random keys from db.expires
    Expire->>JEM: Deletes expired keys immediately and reclaims RAM

    Client->>Server: Heavy write query arrives (RAM usage greater than  maxmemory)
    Server->>Evic: performEvictions(): Sizing eviction pool
    Evic->>Evic: Samples N random keys (default 5) into 16-element Eviction Pool
    Evic->>Evic: Sorts pool by idle time (LRU) or logarithmic counter (LFU)
    Evic->>JEM: Evicts best candidate key using dbAsyncDelete (UNLINK)
    JEM-->>Server: Physical memory arena reclaimed
    Server->>Client: Executes incoming write query safely

Pathway C: Trace of Fork Copy-on-Write (COW) Persistence¶

sequenceDiagram
    autonumber
    participant Main as "Redis Main Process (PID 1000)"
    participant Linux as "Linux OS Kernel"
    participant Child as "Child Process (PID 1001)"
    participant Disk as "Physical Disk (dump.rdb)"

    Main->>Linux: fork() system call invoked by BGSAVE
    Linux->>Linux: Duplicates page table pointers (Zero memory copy!)
    Linux-->>Child: Spawns Child Process with identical read-only memory view
    Child->>Disk: Sequentially iterates keyspace and writes dump.rdb

    Note over Main,Linux: Incoming high-throughput writes arrive at Main Process:
    Main->>Linux: Modifies memory page 42
    Linux->>Linux: MMU triggers Page Fault to Allocates NEW 4KB physical page
    Linux->>Linux: Copies original page 42 bytes to new page for Main Process
    Note over Main,Linux: Child process continues reading original page 42 without locking!
    Child->>Disk: Finalizes dump.rdb and issues fsync()
    Child-->>Main: Exits with code 0 (SIGCHLD)

3. Quick Revision Flashcard Matrix ("Connect the Cause to the Effect")¶

If this "Dot" changes / breaks...	Which connected "Dot" immediately suffers?	Why? (The Mechanical Link)
*A developer runs `KEYS ` or `HGETALL` on a 1M element hash**	All connected clients & Sentinel health-checks	Single-threaded execution means no other socket or timer event can be processed until the \(O(N)\) command completes. Sentinel detects timeout and triggers false failovers.
`transparent_hugepage` is enabled in Linux kernel	Memory amplification during `BGSAVE`	When the main thread writes to memory during fork COW, the OS must copy an entire 2MB hugepage instead of a fine-grained 4KB page, doubling or tripling memory consumption and triggering OOM kills.
A replica falls behind by more than `repl-backlog-size`	Primary server disk I/O & Network	Partial resynchronization (`PSYNC2`) fails. Primary is forced to fork an entire full RDB snapshot (`BGSAVE`), ship the multi-gigabyte file over the wire, and force the replica to wipe its memory and reload.
`client-output-buffer-limit slave` is set too low	Replication stability	High write throughput fills the replica's output buffer faster than the network can drain it. Redis disconnects the replica, throwing the cluster into an infinite full-resync disconnect loop.
A string key is appended repeatedly past 44 bytes	Memory allocation efficiency	Redis dynamically flips object encoding from `OBJ_ENCODING_EMBSTR` (single allocation of `robj` + SDS) to `OBJ_ENCODING_RAW` (two separate `malloc` allocations), increasing memory fragmentation and pointer chasing.
`appendfsync` is set to `always`	Write throughput (\(100\text{k req/s} \rightarrow 1\text{k req/s}\))	The main execution thread blocks on every single write command until the physical disk drive controller flushes the byte to non-volatile storage via `fsync`.
A Hash table expands without progressive rehashing	Query latency spikes	Reallocating a 10-million key hash table in a single blocking step would freeze the server for seconds. Redis solves this by progressively moving 1 bucket per user query (`dictRehashStep`).
`maxmemory-policy` is left as default `noeviction`	Application writes	Once RAM reaches `maxmemory`, all write operations (`SET`, `HSET`, `LPUSH`) fail immediately with `OOM command not allowed when used memory > 'maxmemory'`. Reads remain functional.