Phase 6: Replica Sets, Consensus Failover & The Oplog Pipeline¶
Audience: Principal Distributed Systems Architect & Site Reliability Engineer
Source References:src/mongo/db/repl/,src/mongo/db/repl/oplog.cpp,src/mongo/db/repl/replication_coordinator_impl.cpp
1. The Oplog Cellular Anatomy (local.oplog.rs)¶
Replication in MongoDB is driven by the Oplog (Operations Log)—a special capped collection located in the local database (local.oplog.rs).
Oplog Capped Circular Buffer on Primary:
┌────────────────────────────────────────────────────────────────────────┐
│ [Oldest Entry: ts: 1716380000] ──► ... ──► [Newest Entry: ts: 1716385000] │
└────────────────────────────────────────────────────────────────────────┘
▲ ▲
│ Once max size reached, overwrites oldest │ Head of active writes
1.1 Anatomy of an Oplog Entry¶
Every mutating operation produces a structured BSON document in local.oplog.rs:
{
"ts": Timestamp(1716382025, 4), // 64-bit: 32-bit seconds, 32-bit increment
"t": NumberLong(3), // Election Term number
"h": NumberLong("8492049182"), // Unique 64-bit operation hash
"v": 2, // Oplog version
"op": "u", // "i" (insert), "u" (update), "d" (delete), "c" (command)
"ns": "ecommerce.orders", // Target database and collection namespace
"ui": UUID("a4d3-e2f1-..."), // Permanent collection UUID
"o2": { "_id": 10842 }, // Query selector for update/delete
"o": { "$v": 2, "$diff": { "u": { "status": "SHIPPED" } } } // Idempotent delta
}
1.2 The Strict Invariant of Idempotency¶
Application commands may be relative (e.g., { $inc: { views: 1 } }).
However, the Oplog transforms relative operations into absolute values:
2. Secondary Replication Pipeline: Fetch, Buffer & Parallel Apply¶
Secondaries do not execute queries through the query planner. Instead, they stream and apply raw Oplog deltas:
┌────────────────────────────────────────────────────────────────────────┐
│ Secondary Node Replication Engine │
├────────────────────────────────────────────────────────────────────────┤
│ 1. OplogFetcher: Tailable Cursor │
│ - Opens long-polling tailable cursor on Primary's local.oplog.rs │
│ - Streams batches of BSON oplog entries over the wire │
│ │
│ 2. OplogBuffer (In-Memory Queue) │
│ - Buffers incoming entries in RAM (default limit: 256 MB) │
│ - Decouples network transfer speed from local disk write speed │
│ │
│ 3. Parallel Batch Applier (SyncTail) │
│ - Groups oplog entries into discrete transactional batches │
│ - Hashes entries by Document _id across worker threads: │
│ Thread 1: Applies mutations to _id: 101, _id: 205 │
│ Thread 2: Applies mutations to _id: 102, _id: 308 │
│ - Guarantees zero locking conflicts between different documents! │
└────────────────────────────────────────────────────────────────────────┘
3. Election Protocol v1 (Raft-like Consensus)¶
MongoDB 3.2+ uses Election Protocol v1, a consensus algorithm providing sub-2-second failover.
Candidate Replica Election State Machine:
1. Heartbeat Loss Detection:
- Replicas ping primary every 2,000ms.
- If no reply for electionTimeoutMillis (default: 10,000ms), replica declares election.
2. Term Increment & Pre-Vote Phase:
- Replica increments term: term++.
- Executes a "Dry Run" Pre-Vote to verify network reachability with majority.
3. Vote Request Broadcast:
- Replica sends vote requests to all voting members.
4. Voter Validation Criteria (Strict Rules):
- Voter will ONLY grant vote if:
a. Candidate's term > Voter's term.
b. Candidate's priority > 0.
c. Candidate's last Oplog timestamp >= Voter's last Oplog timestamp.
(A stale replica can NEVER be elected!)
d. Voter has not already voted for another candidate in this term.
5. Leadership Assumption:
- If candidate secures votes from strict majority: floor(N/2) + 1.
- Immediately steps up as PRIMARY and writes a "no-op" entry to local.oplog.rs.
4. The Rollback Mechanism & The rollback/ Directory¶
4.1 When Does a Rollback Occur?¶
Consider a network partition where Primary A is isolated with Client 1:
1. Client 1 executes insertOne({ name: "Uncommitted" }) with { w: 1 }.
2. Primary A commits to its local Oplog, but the write never reaches the majority of secondaries.
3. Meanwhile, the remaining secondaries form a majority, elect Replica B as new Primary, and continue writing.
4. The partition heals: Old Primary A reconnects to the replica set.
5. Node A discovers that Node B has a higher term and different Oplog entries.
6. Node A cannot simply delete its history because client data would vanish without trace.
Rollback Reconciliation Algorithm:
1. Node A transitions immediately to RECOVERING state.
2. Locates Common Ancestor:
- Scans backward through its Oplog until it finds an entry matching Node B's Oplog.
3. Unwinds Divergent Operations:
- Extracts all local operations executed after the common ancestor.
4. Serializes Divergent Documents to Disk:
- Writes the deleted/altered documents as BSON files in:
<dbpath>/rollback/<collection_uuid>/
5. Resets Local Oplog & Catches Up:
- Truncates divergent Oplog entries.
- Fetches missing entries from the new Primary B.
- Returns to SECONDARY state.
Administrative Recovery:¶
Engineers can inspect and manually re-inject rolled-back documents using bsondump: