Phase 7: JVM, Garbage Collection, Off-Heap Memory & Kernel Tuning¶
Audience: Principal Systems Architect & Performance Tuning Specialist
Source References:conf/jvm.options,conf/jvm11-server.options,src/java/org/apache/cassandra/utils/memory/
1. Dual-Tier Memory Architecture: Heap vs. Off-Heap¶
Cassandra utilizes a hybrid memory model dividing memory between the managed Java Virtual Machine (JVM) heap and unmanaged native off-heap memory:
Total Host Physical RAM (e.g. 64 GB)
┌────────────────────────────────────────────────────────────────────────┐
│ JVM Heap Memory (Typically 16 GB to 31 GB) │
│ - Compressed Ordinary Object Pointers (Compressed OOPs: < 32GB) │
│ - Row mutation objects, query AST parsing, internal coordinators │
│ - Key Cache (Partition key row pointers) │
├────────────────────────────────────────────────────────────────────────┤
│ Direct Off-Heap Native Memory (JEMalloc / Direct ByteBuffers: ~20 GB) │
│ - Off-Heap Memtables (If configured via offheap_buffers) │
│ - Bloom Filters (Filter.db loaded directly outside JVM heap) │
│ - CompressionInfo metadata (LZ4/Snappy chunk offsets) │
│ - Netty network socket buffers (Zero-copy network transfers) │
├────────────────────────────────────────────────────────────────────────┤
│ Linux OS Page Cache (~13 GB) │
│ - Caches compressed SSTable physical disk blocks │
└────────────────────────────────────────────────────────────────────────┘
1.1 The 32 GB Compressed OOPs Boundary¶
In 64-bit JVMs, pointers are 8 bytes wide. When heap size is configured below 32 GB (typically 31 GB), the HotSpot JVM enables Compressed OOPs (Ordinary Object Pointers), representing pointers as 32-bit relative offsets:
- Yields a 30% reduction in JVM memory footprint.
- Setting heap to 33 GB forces 64-bit pointers, wasting gigabytes of RAM on padding!
2. JVM Garbage Collection Strategy: G1GC vs. ZGC¶
JVM Garbage Collection pauses are the #1 cause of false node convictions in Gossip, dropped mutations, and 5-second P99 latency spikes.
2.1 Production G1GC Tuning (jvm11-server.options)¶
G1GC partitions heap into thousands of equal-sized memory regions:
# Production G1GC Flags for Cassandra
-XX:+UseG1GC
-XX:G1RSetUpdatingPauseTimePercent=5
-XX:MaxGCPauseMillis=200
-XX:InitiatingHeapOccupancyPercent=45
-XX:G1ReservePercent=15
-XX:G1HeapRegionSize=16m
InitiatingHeapOccupancyPercent=45: Starts concurrent marking when the heap is 45% full. Setting this too high (e.g., 70%) triggers catastrophic stop-the-world Full GC pauses when memory allocation surges!
- G1ReservePercent=15: Reserves 15% of heap as emergency buffer to prevent "to-space exhausted" evacuation failures.
2.2 Next-Generation Ultra-Low Latency: Generational ZGC (Java 17/21)¶
In Cassandra 4.1/5.0 on modern Java runtimes:
- ZGC performs reference coloring and load barriers to execute marking, relocation, and compaction concurrently with application threads. - Result: GC pauses are guaranteed under 1 millisecond even on 128 GB heaps!3. Host Linux Kernel & Hardware Tuning¶
3.1 Kernel Parameters (/etc/sysctl.conf)¶
# Completely disable or minimize memory swapping
vm.swappiness = 1
# Increase max memory map count for thousands of SSTables
vm.max_map_count = 1048575
# Expand network connection queue for Cassandra CQL binary traffic
net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 4096
# Allocate adequate dirty page cache flush behavior
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10