Phase 3: Storage Engine: CommitLog, Memtable & SSTable Anatomy¶

Audience: Principal Storage Systems Architect & Performance Systems Engineer
Source References: src/java/org/apache/cassandra/db/commitlog/, src/java/org/apache/cassandra/db/memtable/, src/java/org/apache/cassandra/io/sstable/

1. The LSM-Tree Write Path Paradigm¶

Cassandra does not modify physical data in place. Instead, it utilizes an LSM-Tree (Log-Structured Merge-Tree) architecture where all writes are strictly sequential append-only operations:

[Incoming Mutation]
       │
       ├──► 1. Append to CommitLog (Sequential Disk Write to NVMe for Durability)
       │
       └──► 2. Insert into Memtable (Sorted In-Memory Data Structure in RAM)
                   │
                   ▼ (When Memtable is Full)
            3. Flush Sequential Immutable SSTable to Disk

2. CommitLog Mechanics: Durability & Segment Management¶

The CommitLog guarantees crash recovery. If power fails, un-flushed Memtables in RAM are reconstructed by replaying the CommitLog.

2.1 Segmented Pre-Allocated Files¶

Location: /var/lib/cassandra/commitlog/
Files are named: CommitLog-<version>-<timestamp>.log.
Default segment size: 32 Megabytes (commitlog_segment_size_in_mb).
When a segment reaches 32 MB, a new segment is allocated. Segments are deleted only after all corresponding Memtables have been flushed to SSTables.

2.2 CommitLog Sync Modes: `periodic` vs. `batch`¶

# 1. Periodic Mode (Default for High-Throughput Production)
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000 # Flushes dirty buffers to disk every 10s

# 2. Batch Mode (Financial / Zero-Loss SLA)
commitlog_sync: batch
commitlog_sync_batch_window_in_ms: 2 # Blocks client write until fsync() finishes

- In periodic mode, writes are appended to an in-memory mapped byte buffer; the OS flushes dirty pages every 10 seconds. In the event of catastrophic server loss, a maximum of 10 seconds of writes can be lost. - In batch mode, Cassandra groups concurrent writes and issues an explicit fsync every 2ms before returning success to the client.

3. Memtable Cellular Internals: SkipLists & Off-Heap Allocators¶

The Memtable is an in-memory sorted cache of mutations for a specific table.

Memtable Data Structure: ConcurrentSkipListMap
Level 3: [Partition A] ──────────────────────────► [Partition K]
Level 2: [Partition A] ─────────► [Partition F] ──► [Partition K]
Level 1: [Partition A] ──► [C] ──► [Partition F] ──► [Partition K]

3.1 Memory Allocation Options¶

In cassandra.yaml: - heap_buffers: Memtables live entirely in the Java Virtual Machine (JVM) young and old generation heaps. High write churn generates intense GC pressure. - offheap_buffers: Cell byte arrays are allocated outside the JVM heap using native C allocators (JEMalloc), eliminating JVM Garbage Collection pauses entirely during write ingestion! - offheap_objects: In addition to cell bytes, row pointers and index structures are allocated in native off-heap memory.

3.2 The Flush Trigger Lifecycle¶

A Memtable flush is triggered when: 1. Total Memtable memory crosses memtable_total_space_in_mb (typically 25% of heap). 2. The CommitLog approaches commitlog_total_space_in_mb. 3. An administrative flush is invoked (nodetool flush).

Flushing Protocol: - The active Memtable is atomically marked READ-ONLY (flushing_memtable). - A brand new empty active Memtable is instantiated instantly to accept incoming writes. - A background FlushWriter thread iterates sequentially through the read-only Memtable and writes an immutable SSTable to disk.

4. The 8-Component Anatomy of an SSTable¶

Once written, an SSTable is strictly immutable. It is never modified, updated, or rewritten in place. An SSTable consists of 8 coordinated physical files on disk:

/var/lib/cassandra/data/keyspace1/users-a82f0910.../
├── nb-1-big-Data.db              <--- Primary Data Storage
├── nb-1-big-Index.db             <--- Partition Index
├── nb-1-big-Summary.db           <--- In-Memory Index Summary
├── nb-1-big-Filter.db            <--- Off-Heap Bloom Filter
├── nb-1-big-Statistics.db        <--- Metadata & Droppable Tombstone Stats
├── nb-1-big-CompressionInfo.db   <--- LZ4 Chunk Offset Ledger
├── nb-1-big-TOC.txt              <--- Table of Contents
└── nb-1-big-Digest.crc32         <--- Checksum Integrity Hash

Detailed Functional Dissection:
1. Data.db:
   The compressed binary byte stream of raw partitions, clustering rows, cells, and tombstones.

2. Index.db (Partition Index):
   Sorted array of partition keys and their exact byte offset boundaries within Data.db.

3. Summary.db (Index Summary):
   Because Index.db can be gigabytes on disk, Cassandra samples Index.db every 128 keys
   (index_interval) and loads this Summary into RAM. A binary search in RAM navigates directly
   to the correct 128-key window in Index.db.

4. Filter.db (Bloom Filter):
   Probabilistic bit array loaded into off-heap memory.
   Before opening any files, Cassandra queries the Bloom filter:
   "Does this SSTable contain Partition Key X?"
   - "NO": 100% guarantee key is not in this SSTable. Skip disk entirely!
   - "YES": Key might be present (~1% false positive). Seek disk.

5. Statistics.db:
   Stores min/max timestamp, TTL distributions, and droppable tombstone ratios used by
   the compaction engine to select candidate SSTables.

6. CompressionInfo.db:
   Maps logical uncompressed data byte offsets to compressed 64KB physical blocks on disk.

5. Bloom Filter Mathematical Tuning¶

The Bloom filter false positive probability is configurable per table:

ALTER TABLE users WITH bloom_filter_fp_chance = 0.01;

5.1 Sizing Equation¶

To achieve false-positive probability $p$ with $n$ keys: $$\text{Filter Size (Bits) } m = -\frac{n \ln p}{(\ln 2)^2}$$ $$\text{Number of Hash Functions } k = \frac{m}{n} \ln 2$$

Setting fp_chance = 0.01 (1% false positive) requires 9.6 bits per key.
Setting fp_chance = 0.1 (10% false positive) requires 4.8 bits per key (saving 50% RAM at the expense of doubling random disk reads).