Phase 1: MongoDB Process Model, Thread Architecture & Memory Allocation¶

Audience: Principal Distributed Systems Architect & Infrastructure Engineer
Source References: src/mongo/db/service_entry_point.cpp, src/mongo/db/storage/wiredtiger/, src/mongo/util/net/

1. Process Architecture: `mongod` vs. `mongos`¶

MongoDB distributed deployments consist of two primary binary daemons:

┌────────────────────────────────────────────────────────────────────────┐
│ mongos (Stateless Query Router Daemon)                                 │
├────────────────────────────────────────────────────────────────────────┤
│ - No local storage engine; completely diskless                         │
│ - Maintains client connection pool                                     │
│ - Caches routing table metadata from Config Servers (CSRS)             │
│ - Scatter-gather query dispatcher: Merges results from shards          │
└────────────────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────────────────┐
│ mongod (Stateful Data Engine Daemon)                                   │
├────────────────────────────────────────────────────────────────────────┤
│ - Houses the WiredTiger storage engine & local disk files              │
│ - Executes query planning, index scans, and document retrieval         │
│ - Manages local concurrency (Intent locks, MVCC, hazard pointers)      │
│ - Drives replication consensus (Oplog generator, Raft-like election)   │
└────────────────────────────────────────────────────────────────────────┘

2. Thread Architecture & Concurrency Model¶

When an application connects to mongod, incoming connections are managed through ServiceEntryPoint (service_entry_point_service_core.cpp):

[Client Connections]
   │    │    │
   ▼    ▼    ▼
[ServiceEntryPoint: Thread-per-Connection or Thread Pool]
   │
   ▼
[Worker Execution Thread]
   ├── 1. Wire Protocol Parsing (OP_MSG BSON decompression)
   ├── 2. Client Authentication & Authorization (SCRAM-SHA-256)
   ├── 3. Lock Acquisition (Acquires IS/IX locks from LockManager)
   ├── 4. Query Execution / Storage Access (Calls WiredTiger API)
   └── 5. Formats BSON response and writes to client socket

2.1 The Two Threading Models: `adaptive` vs. `synchronous`¶

Configurable via net.serviceExecutors: 1. synchronous (Thread-per-Connection): - Spawns one dedicated OS thread per open client connection. - At 10,000 connections, 10,000 OS threads thrash the Linux kernel scheduler with massive stack memory overhead (\(10,000 \times 1\text{MB stack} = 10\text{GB RAM}\)). 2. adaptive (Thread Pool Architecture): - Uses an internal work queue and a dynamic worker thread pool (typically \(2 \times \text{CPU cores}\)). - Decouples client network socket connections from OS execution threads, preventing thread starvation during high connection surges.

3. Cellular Memory Allocation: WiredTiger Cache vs. OS Page Cache¶

A fundamental architectural principle of MongoDB is the Dual-Cache Model:

Total Host Physical RAM (e.g., 64 GB)
┌────────────────────────────────────────────────────────────────────────┐
│ WiredTiger Internal Memory Cache (50% RAM - 1GB = 31.5 GB)             │
│ - Uncompressed in-memory B-Tree pages (wt_page_t)                      │
│ - Dynamic update chains (wt_update_t)                                  │
│ - Hazard pointer tracking arrays                                       │
│ - WiredTiger transaction rollback history                              │
├────────────────────────────────────────────────────────────────────────┤
│ Operating System Page Cache & Buffers (~28 GB)                         │
│ - Compressed disk blocks (Snappy/Zstd compressed pages cached by OS)   │
│ - Zero double-caching penalty: RAM stores compressed OS cache +        │
│   uncompressed active WT working set                                   │
├────────────────────────────────────────────────────────────────────────┤
│ mongod Process Overhead (~4.5 GB)                                      │
│ - Connection stacks (1MB per thread in synchronous mode)               │
│ - In-memory sort buffers (allowDiskUse: false limit 100MB per query)   │
│ - Aggregation pipeline memory buffers                                  │
└────────────────────────────────────────────────────────────────────────┘

3.1 WiredTiger Cache Calculation Formula¶

/* WiredTiger default cache calculation */
cache_size_gb = std::max(0.5 * (total_ram_gb - 1.0), 0.256);

Production Configuration:¶

storage:
  wiredTiger:
    engineConfig:
      cacheSizeGB: 30

4. Linux Kernel Tuning Matrix for MongoDB¶

Running MongoDB with default Linux settings can cause catastrophic multi-second lockups:

Kernel Setting	Production Target	Failure Mode if Not Tuned
`transparent_hugepage`	`never`	2MB memory page allocation forces compaction stalls; WiredTiger cache eviction freezes for 100ms+
`vm.swappiness`	`1` (or `0`)	Linux swaps active WiredTiger cache pages to swap space, collapsing read latency
`vm.max_map_count`	`262144`	High number of WiredTiger data files and collections causes `mmap` allocation failure
`ulimit -n (open files)`	`64000`	Hits `EMFILE` error; drops client connections and fails shard replication
`ulimit -u (max processes)`	`64000`	Cannot spawn worker threads under high connection spikes
`numactl --interleave=all`	Mandatory on NUMA	Memory allocation becomes unbalanced across NUMA nodes, causing remote memory access latency

# Production Kernel Tuning Script
sudo sysctl -w vm.swappiness=1
sudo sysctl -w vm.max_map_count=262144
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

# Launch mongod with NUMA interleaving
numactl --interleave=all /usr/bin/mongod --config /etc/mongod.conf