Skip to content

Load Shedding

1. Decision

Implement Load Shedding at the API Gateway and service entry points to drop low-priority, non-essential incoming requests when CPU, RAM, or connection pool utilization crosses critical thresholds, protecting core system health and preserving capacity for high-priority traffic.

2. 30-Second Answer

When a service experiences sudden, unexpected traffic spikes beyond its maximum capacity, CPU queues fill up, latency skyrockets exponentially, and the service enters a Livelock State (spending 100% of CPU time switching thread contexts and failing every request). Load Shedding is a active defense mechanism: when hardware resources (CPU, Memory, In-flight concurrency) cross safety thresholds (e.g. \(>85\%\) CPU usage), the service intentionally rejects incoming low-priority requests immediately with HTTP 503 Service Unavailable or 429 Too Many Requests. By shedding 20% of excess traffic at the front door, the service processes the remaining 80% of high-priority traffic successfully at normal SLAs, preventing total cluster collapse.

3. Decision Matrix

Option Best For Avoid When Latency Scalability Consistency Operational Complexity
Load Shedding (CPU / Concurrency Adaptive Drop) High-traffic public platforms (Coachella ticket sales, Black Friday e-commerce), microservices prone to flash traffic spikes Low-volume systems where auto-scaling handles all traffic bounds Ultra-Low (Rejects dropped requests immediately in 0ms) Protects system availability under \(10\times\) overload spikes Rejects excess traffic to protect processing state Medium
Traditional Auto-Scaling Only (HPA) Predictable, gradual traffic growth patterns (e.g. daily morning usage increases) Instantaneous flash traffic surges (auto-scaling takes 2-5 minutes to spin up new pods) Low during normal scale; High during flash spike before pods boot High for gradual traffic curves Standard processing Low

4. Selection Rules

Requirement Best Choice Rationale / Example
Survive Sudden 10x Flash Traffic Spike Adaptive Load Shedding Drop low-priority traffic instantly when CPU \(>85\%\), allowing high-priority payment calls to succeed
Prioritize Authenticated Premium Users over Guests Priority-Based Load Shedding Shed requests from unauthenticated guests (Priority=Low) while serving VIP users (Priority=High)
Protect Core Processing from Background Webhooks Criticality-Based Load Shedding Reject background analytical webhooks before rejecting interactive checkout API requests
Congestion Control via Queue Delay (CoDel / Little's Law) Adaptive In-Flight Concurrency Drop requests if queue wait time exceeds 100ms using Little's Law (\(\text{Concurrency} = \text{Arrival Rate} \times \text{Latency}\))

5. Tradeoffs

  1. System Survival vs Immediate Request Rejection
  2. Gain: 80% of users complete operations cleanly; service remains online and stable.
  3. Lose: 20% of users receive immediate HTTP 503 / 429 error responses.
  4. Instant zero-latency Rejection vs Queueing Latency
  5. Gain: Dropping requests immediately in 0ms frees CPU and RAM resources instantly.
  6. Lose: Users must manually retry or wait for traffic spikes to subside.
  7. Adaptive Concurrency Metrics vs Hard Static Quotas
  8. Gain (Adaptive): Dynamically adjusts concurrency limits based on live CPU/RAM and response latency.
  9. Lose (Adaptive): Requires tuning algorithm parameters (e.g., Netflix CoDel / TCP Vegas limits).
  10. Priority Tier Classification Overhead
  11. Gain: Guarantees high-value business transactions (Payment) succeed during traffic overload.
  12. Lose: Requires classifying all incoming API endpoints into explicit Priority Tiers (CRITICAL, HIGH, LOW).
  13. Load Shedding vs Rate Limiting
  14. Gain (Load Shedding): Rejects traffic based on internal server health (CPU/RAM), regardless of client identity.
  15. Lose (Load Shedding): Affects legitimate users if system resources are exhausted.

6. Interview Counter Questions

  1. What is the fundamental difference between Rate Limiting, Throttling, and Load Shedding?
  2. What is Livelock / Collapse Under Load and how does Load Shedding prevent it?
  3. How does Netflix implement Adaptive Concurrency Limits for Load Shedding?
  4. How do you classify API traffic into Priority Tiers (Critical, High, Normal, Low) for Load Shedding?
  5. What role does Little's Law (\(\text{Concurrency} = \text{Arrival Rate} \times \text{Latency}\)) play in Load Shedding?
  6. Why is Auto-Scaling alone insufficient to protect against sudden flash traffic spikes?
  7. What HTTP status code should a server return when shedding load, and what headers should be included?
  8. How does CoDel (Controlled Delay) algorithm shed load based on queue wait times?
  9. Where in the infrastructure stack should Load Shedding be applied (Edge Gateway vs Individual Microservice)?
  10. How do you test Load Shedding mechanisms in a staging environment?

7. Gold Standard Answers

  1. Rate Limiting: Restricts requests based on client identity (IP / API Key) quotas. Throttling: Delays or caps execution speed. Load Shedding: Drops requests based on server health (CPU/RAM) to prevent system crash.
  2. Livelock occurs under heavy load when CPU spends 100% of its time context switching, allocating memory, and timing out requests, delivering 0 QPS throughput. Load Shedding drops excess requests in 0ms, restoring full throughput.
  3. Netflix open-sourced concurrency-limits. It measures live RTT latency. If latency increases (indicating queue building), it dynamically lowers the max allowed concurrency limit using TCP congestion algorithms (Vegas/Additive Increase Multiplicative Decrease).
  4. Classify requests: (1) CRITICAL: Checkout, Payment, Auth, (2) HIGH: Product details, (3) NORMAL: Search, (4) LOW: Recommendations, Analytics. When CPU \(>85\%\), drop LOW first, then NORMAL.
  5. Little's Law: \(L = \lambda W\). If average latency (\(W\)) doubles due to backend slowness, allowed concurrency (\(L\)) doubles for the same arrival rate (\(\lambda\)). If pool capacity is fixed, queue wait times blow up; Load Shedding caps \(L\) to keep \(W\) low.
  6. Auto-scaling is slow: provisioning new EC2 instances or K8s pods takes 1 to 5 minutes. A 10x traffic spike arrives in seconds, crashing existing pods before auto-scaled pods become ready.
  7. Return HTTP 503 Service Unavailable or HTTP 429 Too Many Requests with a Retry-After: <seconds> header (instructing clients to back off using random jitter).
  8. CoDel measures how long requests wait in execution queues. If minimum queue delay exceeds a target (e.g. 5ms) over an interval, CoDel drops incoming requests to clear queue backlog.
  9. Apply at both layers: (1) API Gateway: Drops low-priority routes globally based on cluster metrics, (2) Individual Microservice: Drops incoming RPCs based on local CPU/RAM/JVM garbage collection pressure.
  10. Run Stress / Load Testing (k6 / Locust / JMeter) to push staging cluster to 200% capacity. Verify CPU caps at 85%, low-priority requests receive 503, and high-priority payment endpoints maintain 100% success rate.

8. Common Mistakes

  • Relying solely on Kubernetes Horizontal Pod Autoscaler (HPA) to survive instant 10x flash traffic surges.
  • Dropping all traffic randomly without implementing Priority Tiers, accidentally dropping high-value payment transactions.
  • Queueing incoming requests infinitely in memory instead of shedding them, causing JVM Out-Of-Memory (OOM) crashes.
  • Returning HTTP 500 Internal Server Error instead of HTTP 503 / 429 with Retry-After headers during load shedding.
  • Implementing load shedding based solely on static hardcoded QPS limits instead of adaptive server health metrics (CPU/RAM/Latency).

9. Keywords

Load Shedding, Livelock, Collapse Under Load, Adaptive Concurrency Limits, Netflix Concurrency Limits, Priority Tiers, Little's Law, CoDel, Controlled Delay, HTTP 503, Retry-After, TCP Vegas, Flash Traffic Spike, Auto-Scaling Latency.

10. Cheat Sheet

  • Core Goal: Intentionally drop low-priority traffic when server is healthy-stressed (CPU \(>85\%\)) to keep core features operating at 100% success.
  • Load Shedding vs Rate Limiting: Rate Limiting = Client Quotas; Load Shedding = Server CPU/RAM Health Defense.
  • Priority Tiers:
  • CRITICAL (Payment/Auth) \(\rightarrow\) Never drop!
  • HIGH (Core Product Data) \(\rightarrow\) Drop last.
  • LOW (Recommendations/Analytics) \(\rightarrow\) Drop FIRST when CPU \(>80\%\).
  • HTTP Response: Return HTTP 503 Service Unavailable + Retry-After: <seconds> header. Rejects in 0ms!
  • Auto-Scaling Failure: Auto-scaling takes 2-5 minutes to boot pods. Load Shedding protects servers in 0 milliseconds during sudden flash spikes.
  • Adaptive Algorithm: Use Adaptive Concurrency Limits (Netflix library) to measure RTT latency and drop requests before queues explode.

11. Related Decisions