Phase 4: The Event Loop & Threaded I/O Architecture¶
Audience: Principal Distributed Systems Architect & Systems Programmer
Source References:src/ae.c,src/ae_epoll.c,src/ae_kqueue.c,src/networking.c,src/server.c
1. The ae Event Loop Multiplexer Abstraction¶
Redis avoids bulky external event libraries (such as libevent or libev) and implements its own ultra-lean \(\approx 1,000\)-line event multiplexer: ae (A simple Event loop).
┌────────────────────────────────────────────────────────┐
│ aeEventLoop (ae.c) │
├────────────────────────────────────────────────────────┤
│ OS Platform Detection at Compile Time: │
│ #ifdef HAVE_EVPORT ===> ae_evport.c (Solaris) │
│ #elif HAVE_EPOLL ===> ae_epoll.c (Linux) │
│ #elif HAVE_KQUEUE ===> ae_kqueue.c (macOS / BSD) │
│ #else ===> ae_select.c (POSIX fallback│
└────────────────────────────────────────────────────────┘
1.1 Why epoll Outperforms select and poll¶
select/poll: \(O(N)\) complexity. Every call copies the entire array of file descriptors (FDs) from user space to the Linux kernel. The kernel iterates through all FDs to detect readiness. At 50,000 connections, CPU stalls.epoll(ae_epoll.c): \(O(1)\) complexity. Uses a kernel red-black tree (epoll_create) and a callback-based ready-list. When a network packet arrives on a socket, the network interface card (NIC) interrupt invokes the kernel callback, placing the ready FD into the ready queue.epoll_wait()returns strictly the active FDs in \(O(1)\) time.
2. File Events vs. Time Events¶
The event loop coordinates two distinct categories of events:
┌────────────────────────────────────────────────────────────────────────┐
│ aeEventLoop Execution │
├────────────────────────────────────────────────────────────────────────┤
│ [File Events] │
│ - Network Socket I/O (Clients, Replicas, Cluster Bus) │
│ - AE_READABLE: Socket has incoming bytes or new connection (accept) │
│ - AE_WRITABLE: Socket send buffer has capacity for outgoing bytes │
├────────────────────────────────────────────────────────────────────────┤
│ [Time Events] │
│ - Periodic background maintenance routines (serverCron) │
│ - Frequency controlled by 'hz' (default 10 = runs every 100ms) │
│ - Active key expiration, progressive rehashing, defragmentation, │
│ AOF fsync triggers, Sentinel heartbeats, cluster gossip PING │
└────────────────────────────────────────────────────────────────────────┘
3. The Grand Execution Pipeline: beforeSleep() & afterSleep()¶
The master loop is driven by aeMain(), which repeatedly executes aeProcessEvents():
┌─────────────────────────────────┐
│ aeMain() Loop │
└────────────────┬────────────────┘
│
▼
[ Call beforeSleep() ]
- Flush AOF buffer to disk (write syscall)
- Fast memory eviction check
- Module beforeSleep callbacks
- Flush client output buffers to sockets
│
▼
[ Call epoll_wait(timeout) ]
- Sleep until socket event arrives
- OR sleep until closest Time Event expires
│
▼
[ Call afterSleep() ]
- Module afterSleep callbacks
- Refresh server time cache
│
▼
[ Dispatch File Events ]
- For each active FD:
- Call readHandler (readQueryFromClient)
- Call writeHandler (sendReplyToClient)
│
▼
[ Dispatch Time Events ]
- Execute serverCron() if deadline reached
│
└───────────────► (Repeat)
The beforeSleep() Optimization¶
Redis does not write replies to sockets immediately upon executing a command. Instead, it formats the reply in RAM and registers the client in a queue (server.clients_pending_write).
In beforeSleep(), Redis flushes all pending client buffers to their respective sockets in a single batch. This minimizes network packet fragmentation, coalesces small payloads into maximum transmission units (MTU), and reduces the number of write() system calls by up to 80%.
4. Threaded I/O Architecture (Redis 6.0+)¶
4.1 The Kernel Network Wall¶
At 100,000+ operations/sec, the single-threaded CPU core becomes 100% saturated not by in-memory data operations, but by:
1. read() syscalls reading thousands of small packets from kernel TCP buffers.
2. Parsing RESP (REdis Serialization Protocol) commands.
3. Formatting RESP responses and issuing write() syscalls.
4.2 The Master-Worker I/O Pipeline¶
Redis 6.0 introduced Threaded I/O (io-threads). Crucially, command execution remains strictly single-threaded, ensuring zero locking overhead on the data store. Only network socket reads and writes are offloaded to worker threads.
PHASE 1: MULTI-THREADED SOCKET READ & PARSE
[Client Sockets]
├── Socket 1 ──► [I/O Thread 1] ──► Read bytes & Parse command into c->argv
├── Socket 2 ──► [I/O Thread 2] ──► Read bytes & Parse command into c->argv
└── Socket 3 ──► [I/O Thread 3] ──► Read bytes & Parse command into c->argv
│
▼ (Thread Barrier: Atomic Counter Sync)
PHASE 2: SINGLE-THREADED MAIN ENGINE EXECUTION
[Main Engine Thread]
├── Executes Command 1: SET key1 val1 (Atomic In-Memory Dict Mutation)
├── Executes Command 2: HSET hash1 f1 (Zero Locks, Zero Race Conditions)
└── Executes Command 3: LPUSH list1 item
│
▼ (Queue output responses in c->buf)
PHASE 3: MULTI-THREADED SOCKET WRITE
├── Client 1 ◄── [I/O Thread 1] ◄── Write responses from c->buf to socket
├── Client 2 ◄── [I/O Thread 2] ◄── Write responses from c->buf to socket
└── Client 3 ◄── [I/O Thread 3] ◄── Write responses from c->buf to socket
4.3 Thread Coordination Without Mutex Locks¶
Redis does NOT use heavy pthread_mutex_t locks to distribute socket connections to worker threads. Instead, it uses an array of per-thread linked lists (io_threads_list[target_thread]) and an atomic counter:
/* Main thread assigns clients round-robin */
int item_id = 0;
listIter li;
listNode *ln;
listRewind(server.clients_pending_read, &li);
while((ln = listNext(&li))) {
client *c = listNodeValue(ln);
int target_id = item_id % server.io_threads_num;
listAddNodeTail(io_threads_list[target_id], c);
item_id++;
}
/* Set atomic pending count to wake up worker threads */
setIOPendingCount();
Worker threads poll their atomic counter. Once active, each thread iterates exclusively through its own private list of clients, reading from sockets and parsing RESP. When all workers finish, the main thread proceeds to execute commands sequentially.
Production Configuration:¶
# Recommended: Set to half the available physical CPU cores
# (e.g., 4 threads on an 8-core instance; leave other cores for main thread & OS)
io-threads 4
io-threads-do-reads yes
io-threads only handles socket writes. Setting io-threads-do-reads yes enables multi-threaded reading and protocol parsing, yielding a 2x-3x throughput increase under high client counts).