Skip to content

Heartbeats

1. What is it? (2-3 lines)

A Heartbeat is a periodic control signal (UDP/TCP packet or HTTP ping) transmitted from a monitored node or process to a monitoring coordinator or peer nodes to signal that the node is alive, responsive, and operating normally.

2. Why does it exist?

Distributed servers can crash, freeze, or suffer network partitions without warning. Without explicit health signaling, clients and cluster coordinators continue sending traffic to dead nodes, causing severe request timeouts and cascading failures. Heartbeats provide rapid automated failure detection.

3. Where is it used?

  • Load Balancers: Health checking backend application instances (AWS ALB, Nginx, HAProxy).
  • Cluster Managers & Orchestrators: Kubernetes (kubelet node status pings), HDFS (NameNode/DataNode heartbeats).
  • Consensus & Database Engines: Raft/Paxos leader heartbeats, ZooKeeper session leases, etcd, MongoDB replica sets.

4. How does it work? (simple flow)

  1. Push Model (Node \(\to\) Coordinator): Node sends periodic HEARTBEAT messages every \(T\) seconds (e.g. 1s) to Coordinator \(\to\) Coordinator updates last_seen_timestamp.
  2. Pull Model (Coordinator \(\to\) Node): Coordinator issues HTTP GET /healthz to Node every \(T\) seconds \(\to\) Node responds 200 OK.
  3. Timeout Evaluation: If no valid heartbeat is received within a threshold window (e.g., \(3 \times T = 3\text{s}\)), the coordinator marks the node as UNHEALTHY / DEAD.
  4. Automated Recovery Action: Coordinator removes node from load balancer pools, triggers Leader Election, or launches a replacement pod (Kubernetes).

5. Advantages

  • Simple, lightweight, and low network bandwidth footprint.
  • Enables rapid automated failover and traffic rerouting within seconds of host failure.
  • Essential for maintaining active cluster membership and session leases.

6. Limitations

  • False Positives: Temporary network blips or long Stop-The-World (STW) JVM Garbage Collection pauses trigger false "node dead" alerts.
  • Fixed Threshold Vulnerability: Static timeout windows fail to adapt to fluctuating network latency (addressed by PHI Accrual Failure Detectors).
  • Scalability Bottleneck: 10,000 nodes pushing heartbeats directly to 1 master node creates \(O(N)\) network bottlenecks.

7. Common interview questions (10)

  1. What is the difference between Push-based Heartbeats and Pull-based Health Checks?
  2. How does a Stop-The-World (STW) Garbage Collection pause trigger false-positive heartbeat failures?
  3. What is the PHI Accrual Failure Detector and how does it calculate adaptive failure probabilities?
  4. How do Kubernetes Liveness, Readiness, and Startup Probes differ?
  5. How do Raft Leaders use Heartbeats to prevent followers from triggering unnecessary elections?
  6. What is the relationship between Heartbeat Intervals (\(T\)) and Failure Detection Timeouts (\(K \times T\))?
  7. How does Lease-based Heartbeating work in ZooKeeper / etcd distributed locks?
  8. How do load balancers avoid thundering herd health-check pings when monitoring 1,000 instances?
  9. What happens if a network partition drops heartbeats in one direction only (Asymmetric Network Failure)?
  10. How do Gossip-based Heartbeats scale health checks across massive decentralized clusters?

8. Common misconceptions

  • Misconception: Passing a network heartbeat means the application is fully functional. Fact: A host OS can send network heartbeats while the application database connection pool is completely dead! (Why deep /healthz checks are required).
  • Misconception: Shorter heartbeat intervals are always better. Fact: Ultra-short intervals (e.g. 50ms) cause massive false-positive failover loops during transient network jitter.

10. 30-second cheat sheet

  • Push: Node \(\to\) Master (I am alive); Pull: Master \(\to\) Node (Are you alive?).
  • Rule of Thumb: Timeout = \(3 \times \text{Interval}\) (e.g. 1s interval, 3s timeout) to absorb transient network jitter.
  • Deep Health Check: /healthz endpoint MUST verify underlying DB connections & disk space, not just OS network stack.
  • K8s Probes: Liveness (restarts container), Readiness (removes from load balancer traffic).