Phase 6: High Availability, Sentinel & Clustering¶

Audience: Principal Distributed Systems Architect
Source References: src/replication.c, src/sentinel.c, src/cluster.c, src/cluster.h

1. Replication Internals & The PSYNC2 Protocol¶

Redis replication is asynchronous by default. The master streams mutating RESP commands to connected replicas while acknowledging client writes immediately.

Master-Replica Replication Stream:
[Client] ──► SET k v ──► [Master] ──► (Acknowledge Client immediately)
                            │
                            ▼
           [Replication Backlog Ring Buffer (repl-backlog-size)]
                            │
                            ▼ (Asynchronous TCP stream)
                        [Replica] ──► Executes command locally

1.1 The Replication Backlog Ring Buffer¶

The master maintains a fixed-size circular ring buffer in RAM (server.repl_backlog):

typedef struct replBacklog {
    char *buf;              /* Circular byte buffer */
    long long size;         /* Configured buffer size (e.g., 256 MB) */
    long long master_offset;/* Global monotonic byte offset */
    long long idx;          /* Current insertion pointer in buffer */
    long long count;        /* Actual data bytes currently stored */
} replBacklog;

1.2 The Partial Resynchronization Handshake (`PSYNC2`)¶

When a replica disconnects (e.g., brief network partition) and reconnects: 1. Replica sends: PSYNC <replid> <offset>. 2. Master inspects its ring buffer: - If the requested offset is still present in the circular buffer: - Master responds: +CONTINUE <replid>. - Master streams only the missing differential byte range. (Partial Resync - zero disk or fork overhead!). - If the requested offset has been overwritten or replid does not match: - Master responds: +FULLRESYNC <replid> <offset>. - Master must fork a child process, generate a full RDB snapshot, and transfer the entire file to the replica.

1.3 Diskless Replication (`repl-diskless-sync`)¶

In environments with slow disks (e.g., cloud EBS volumes):

repl-diskless-sync yes
repl-diskless-sync-delay 5

Instead of writing the RDB snapshot to local disk first, the master forks a child that writes the RDB stream directly to replica network sockets, bypassing disk I/O entirely.

2. Redis Sentinel: Quorum, Failover & Split-Brain Mitigation¶

Redis Sentinel is an active distributed consensus layer that monitors master-replica topologies.

Sentinel Topology & Quorum:
         [Sentinel 1] ◄── Gossip (Pub/Sub) ──► [Sentinel 2]
              │ \                                 / │
              │  \                               /  │
              │   ▼                             ▼   │
              │    [Master] ◄── Replication ── [Replica 1]
              │                                     │
              ▼                                     ▼
         [Sentinel 3] ─────────────────────────► [Replica 2]

2.1 The Two-Phase Failure Detection State Machine¶

Subjective Down (SDOWN):
Sentinel sends PING every second.
If the master fails to respond with +PONG within down-after-milliseconds (e.g., 5000ms), Sentinel flags the master as SDOWN.
Objective Down (ODOWN):
The Sentinel queries other Sentinels via SENTINEL is-master-down-by-addr.
If at least Quorum (e.g., 2 out of 3) Sentinels confirm SDOWN, the master enters ODOWN.

2.2 Leader Election & Failover (Raft-like Epoch Voting)¶

The detecting Sentinel increments the configuration epoch: current_epoch++.
It requests authorization to become the failover leader.
If it secures votes from a majority of Sentinels ($\lfloor N/2 \rfloor + 1$), it is elected Leader.
The Leader selects the best replica using ranking criteria:
Lowest replica-priority (0 means never promote).
Highest replication_offset (most up-to-date with master).
Lexicographically smallest runid.
Promotes replica via SLAVEOF NO ONE and reconfigures remaining nodes via SLAVEOF <new_master_ip>.

2.3 The Split-Brain Disaster & Prevention¶

If a master becomes network-isolated from Sentinels and replicas, but remains reachable by a subset of application clients: - Sentinels failover to a replica. - Clients continue writing to the isolated old master. - When the partition heals, the old master demotes to a replica, and all data written to it during the partition is permanently wiped!

Fencing Split-Brain with Kernel Invariants:¶

min-replicas-to-write 1
min-replicas-max-lag 10

Mechanism: The master tracks TCP ACKs from replicas. If fewer than 1 replica acknowledges within 10 seconds, the master immediately rejects all incoming write commands with READONLY You can't write against a read only replica, preventing split-brain data corruption.

3. Redis Cluster: 16,384 Hash Slots & Gossip Protocol¶

Redis Cluster provides horizontal scale-out across multiple master nodes without a central proxy.

Redis Cluster Topology (16,384 Slots Distributed):
┌───────────────────────────┐    ┌───────────────────────────┐    ┌───────────────────────────┐
│ Master A                  │    │ Master B                  │    │ Master C                  │
│ Slots: 0 - 5460           │◄──►│ Slots: 5461 - 10922       │◄──►│ Slots: 10923 - 16383      │
│ Bus Port: 16379           │    │ Bus Port: 16380           │    │ Bus Port: 16381           │
└─────────────┬─────────────┘    └─────────────┬─────────────┘    └─────────────┬─────────────┘
              │ Replication                    │ Replication                    │ Replication
              ▼                                ▼                                ▼
       [Replica A1]                     [Replica B1]                     [Replica C1]

3.1 The Hash Slot Algorithm & Hash Tags¶

Redis partitions the key space into exactly 16,384 slots: $$\text{Slot} = \text{CRC16}(\text{key}) \pmod{16384}$$

Hash Tags (`{...}`)¶

To perform multi-key transactions or atomic Lua scripts across multiple keys, the keys must reside on the same cluster node. Hash tags force hashing on a specific substring: - SET {user:100}:profile "data" - SET {user:100}:orders "[1, 2]" - Hashing evaluates strictly {user:100}. Both keys are guaranteed to land on the identical hash slot and node!

3.2 Client Redirections: `MOVED` vs. `ASK`¶

                                  Client issues: GET key
                                             │
                                             ▼
                                  Calculates CRC16(key) % 16384
                                             │
                       Does this node own the target hash slot?
                                     /             \
                                  YES               NO
                                  /                   \
                   Executes command & returns data.    Is slot undergoing migration?
                                                        /             \
                                                     YES               NO
                                                     /                   \
                                           Returns -ASK node:port    Returns -MOVED node:port
                                          (Temporary 1-shot redue)  (Update client slot cache)

-MOVED <slot> <ip:port>:
Permanent slot ownership change.
Smart client updates its internal routing table mapping that slot to the target node.
-ASK <slot> <ip:port>:
Temporary transition state during online slot resharding.
Client sends ASKING followed by the query to the destination node. The client does NOT update its global slot cache.

3.3 The Cluster Gossip Protocol¶

Nodes communicate over a dedicated Cluster Bus (Client Port + 10000, e.g., 16379). - Every second, a node pings a random subset of nodes with a binary PING packet. - The packet contains a gossip section with status, epoch, and state flags for other known nodes in the cluster. - Failure detection: A node transitions from PFAIL (Possible Fail) to FAIL once a majority of masters broadcast PFAIL flags within cluster-node-timeout.