Phase 9: Real-World Choke Points, Capacity Planning & Telemetry¶

Audience: Principal Database Reliability Engineer & Production Operations Specialist
Source References: src/java/org/apache/cassandra/concurrent/, src/java/org/apache/cassandra/metrics/

1. Production Choke Points & Root Cause Pathologies¶

1.1 The Large Partition Disaster (> 100 MB)¶

Cassandra warns when a partition crosses 100 Megabytes:

WARN  [CompactionExecutor:3] 2026-10-15 09:40:12 CompactionTask.java:240 - 
Writing large partition ecommerce/orders:user_id=108420 (148.520MiB) to sstable nb-4-big-Data.db

Why Large Partitions Destroy Performance:¶

JVM Heap Pressure: During compaction, Cassandra must read the entire partition into memory to reconcile clustering keys and columns. A 2GB partition easily causes an Out-Of-Memory (OOM) OutOfMemoryError: Java heap space crash!
Read Latency Degrades: A query must seek through tens of thousands of clustering rows.
Repair Failure: Streaming a massive partition across the network frequently times out, causing nodetool repair to fail.

Mitigation:¶

Introduce a bucketing salt to the partition key (e.g. (user_id, YYYY_MM)).

2. Thread Pool Telemetry & Backpressure (`nodetool tpstats`)¶

Cassandra organizes operations into specialized thread pools (Stage). Monitoring thread pool saturation is the primary diagnostic technique for Cassandra health:

nodetool tpstats

Sample Output from nodetool tpstats:
Pool Name                    Active   Pending      Completed   Blocked  All time blocked
MutationStage                     8         0      849201940         0                 0
ReadStage                        16         2      194029148         0                 0
CompactionExecutor                2         5         194821         0                 0
FlushWriter                       1         0          84910         0                 0
HintsDispatcher                   0         0           4810         0                 0

Message type           Dropped
READ                         0
MUTATION                   482 <--- CRITICAL ALERT: Writes being dropped!
_TRACE                       0

2.1 Diagnosing Dropped Mutations¶

Pending Tasks: High pending tasks in MutationStage means requests are queuing faster than the node can write to the CommitLog and Memtable.
Dropped Mutations: When queues overflow their capacity, Cassandra drops write mutations! The coordinator receives a timeout and returns WriteTimeoutException to the client.
Root Causes:
Slow disk I/O on NVMe/SSD holding up CommitLog flushes.
Long JVM GC pause freezing all execution threads.

3. Essential Nodetool Diagnostics Playbook¶

3.1 Analyzing Table-Level Latencies (`nodetool tablehistograms`)¶

nodetool tablehistograms ecommerce orders

Outputs granular percentile distributions (P50, P75, P95, P98, P99, Min, Max): - SSTables per Read: Number of physical SSTables touched per read query. - Target: 1.0 to 2.0 SSTables. - If \(> 10\): Compaction is lagging or table schema requires tuning! - Partition Size Distribution: Min/Max size of partitions in bytes. - Read / Write Latency: Microsecond latency percentiles at the storage engine level.

3.2 Inspecting Active Compactions (`nodetool compactionstats`)¶

nodetool compactionstats

Shows: - Number of active compactions currently running. - Progress percentage and remaining bytes. - Pending compaction tasks. (If pending \(> 50\), disk I/O is falling behind write ingestion).

# Throttle or accelerate compaction throughput dynamically (in MB/s)
nodetool setcompactionthroughput 128