Phase 9: Real-World Choke Points, Capacity Planning & Engine Diagnostics¶
Audience: Principal Database Reliability Engineer & Production Operations Specialist
Source References:src/mongo/db/commands/server_status.cpp,src/mongo/db/commands/current_op.cpp
1. Real-World Production Choke Points¶
1.1 The Dirty Cache Saturation Crisis¶
The single most common cause of sudden multi-second MongoDB outages is dirty page saturation in the WiredTiger cache.
WiredTiger Dirty Cache Degradation Stages:
1. Normal Phase (< 5% dirty):
- Background eviction server idles; writes absorb in RAM with sub-millisecond response.
2. Background Eviction Phase (5% to 20% dirty):
- Eviction server wakes up; aggressively writes dirty pages to disk in background.
3. Thread Enlistment Catastrophe (> 20% dirty):
- Background threads cannot keep pace with disk write throughput.
- Application worker threads are forcibly co-opted to perform synchronous disk writes.
- Latency jumps from 1ms to 5,000ms!
Diagnostic Script:¶
// Check dirty cache percentage in real-time
let status = db.serverStatus().wiredTiger.cache;
let dirtyBytes = status["tracked dirty bytes in the cache"];
let maxBytes = status["maximum bytes configured"];
let dirtyPercent = (dirtyBytes / maxBytes) * 100;
print(`WiredTiger Dirty Cache: ${dirtyPercent.toFixed(2)}%`);
1.2 Oplog Window Depletion & Stale Replicas¶
The Oplog Window is the duration of time represented between the oldest and newest entries in local.oplog.rs:
$\(\text{Oplog Window (Hours)} = \frac{\text{Oplog Allocated Size (GB)}}{\text{Write Ingestion Rate (GB/Hour)}}\)$
- If a secondary goes offline for network maintenance for 6 hours, but your Oplog Window is only 4 hours, the secondary becomes permanently stale (
RECOVERING). - It can never catch up using incremental replication and requires a full manual initial sync!
Dynamic Oplog Resizing (Zero Downtime):¶
2. Production Telemetry & Diagnostics Playbook¶
2.1 Deep Analysis of db.serverStatus()¶
const s = db.serverStatus();
// 1. Check WiredTiger Cache Working Set Health
const wt = s.wiredTiger.cache;
const cacheUsed = (wt["bytes currently in the cache"] / wt["maximum bytes configured"]) * 100;
const pagesReadIntoCache = wt["pages read into cache"];
const pagesEvicted = wt["pages written from cache"];
// 2. Check Connection Pool Pressure
const conns = s.connections;
print(`Active: ${conns.current}, Available: ${conns.available}`);
// 3. Check Network & Lock Queues
const globalLock = s.globalLock;
print(`Current Lock Queue - Readers: ${globalLock.currentQueue.readers}, Writers: ${globalLock.currentQueue.writers}`);
2.2 Finding & Killing Stuck Queries (db.currentOp())¶
// Find all queries running for longer than 5 seconds holding locks
db.currentOp({
"active": true,
"secs_running": { "$gt": 5 },
"op": { "$in": [ "query", "update", "remove" ] }
});
// Terminate a pathological rogue query instantly
db.killOp(1084291);
2.3 The explain("executionStats") Golden Metric¶
When tuning slow queries, evaluate the Examined-to-Returned Ratio: $\(\text{Efficiency Ratio} = \frac{\text{totalDocsExamined}}{\text{nReturned}}\)$
- Ratio = 1.0: Perfect index coverage. Every document examined was returned to the client.
- Ratio > 1,000: Catastrophic unindexed filter! The engine examined thousands of documents merely to return a handful of records.
executionStages.stage: Must be IXSCAN or SBE. If COLLSCAN, immediate index required!
2. totalKeysExamined: Number of index keys traversed.
3. totalDocsExamined: Number of documents fetched from disk.
4. executionTimeMillis: Total execution time.
3. Real-Time CLI Diagnostics¶
# High-frequency operational monitoring (Updated every 1 second)
# Displays: insert/query/update rates, dirty cache %, resident RAM, network traffic
mongostat --host 127.0.0.1:27017 1
# Collection-level time tracking
# Pinpoints exactly which collections are consuming CPU time for reads and writes
mongotop --host 127.0.0.1:27017 1