Skip to content

Push vs Pull Metrics

1. Decision

Choose Pull-Based Metrics (Prometheus) as the default architecture for long-lived, auto-discovered microservices and Kubernetes infrastructure to centralize scraping control and decouple target health from metric ingestion; choose Push-Based Metrics (StatsD, Datadog, OpenTelemetry Collector) for short-lived ephemeral serverless jobs (AWS Lambda, batch scripts), event-driven edge workloads, and multi-tenant security environments behind strict firewalls.

2. 30-Second Answer

Metric collection architectures dictate how telemetry moves from applications to monitoring servers. Pull-Based Architecture (Prometheus style) has the monitoring server periodically scrape (fetch HTTP /metrics) from target endpoints. The monitoring server controls scrape frequencies, auto-discovers active target instances via service discovery (Kubernetes API), and immediately detects down instances when scrapes fail. However, Pull requires targets to expose open HTTP metrics ports and struggles with short-lived jobs that terminate before a scrape occurs. Push-Based Architecture (StatsD / Datadog / OTel Collector style) has application instances proactively transmit metrics outbound to a centralized collector. Push excels for ephemeral serverless functions (AWS Lambda) and restricted firewalled environments, but risks overwhelming the metrics backend during traffic spikes unless buffered by local agent collectors.

3. Decision Matrix

Option Best For Avoid When Latency Scalability Consistency Operational Complexity
Pull-Based Metrics (Prometheus / VictoriaMetrics) Long-lived microservices, Kubernetes clusters, infrastructure monitoring, central scrape rate control Short-lived ephemeral jobs (<15s execution e.g. Lambda), firewalled edge devices with inbound port blocks Ultra-Low (Scraped on fixed 15s/30s intervals) High (Scales via Prometheus Federation / Thanos / VictoriaMetrics) Centralized, deterministic scrape health tracking Low to Medium
Push-Based Metrics (StatsD / Datadog / OTel Push) Short-lived serverless functions (AWS Lambda), batch cron jobs, mobile apps, multi-tenant IoT edge devices behind NAT Massive high-volume microservice fleets without local proxy agent buffering (Risks DDOSing collector) Real-time immediate push High (Buffered via local DaemonSet proxy agents e.g. OTel Collector) Eventual push delivery Low (App side); Medium (Collector side)

4. Selection Rules

Requirement Best Choice Rationale / Example
Kubernetes Microservices Infrastructure Monitoring Pull-Based (Prometheus) Prometheus auto-discovers K8s pods via K8s API and scrapes /metrics endpoints
Short-Lived Ephemeral Serverless Tasks (AWS Lambda) Push-Based / Pushgateway Lambda pushes metrics to CloudWatch / OTel Collector before function execution terminates
Instant Target Down/Outage Detection Pull-Based (Prometheus up metric) If Prometheus failed to scrape target pod, up == 0 alerts on instance crash instantly
Strict Security Rules Blocking Inbound HTTP Ports Push-Based Firewalled edge nodes push outbound metrics over HTTPS/gRPC to central collector
Prometheus Ephemeral Job Adapter Prometheus Pushgateway Ephemeral batch jobs push metrics to Pushgateway, which Prometheus scrapes on schedule

5. Tradeoffs

  1. Centralized Health Control vs Application Push Coupling
  2. Gain (Pull): Prometheus controls scrape frequency; knows instantly if an application pod crashed (up = 0).
  3. Lose (Pull): Targets must run an HTTP server exposing an un-authenticated /metrics endpoint.
  4. Ephemeral Batch Job Compatibility vs Longevity Requirements
  5. Gain (Push): Serverless functions push metrics before dying without waiting for a scrape interval.
  6. Lose (Push): If an application pod crashes silently, a Push architecture cannot distinguish between "No Traffic" vs "Instance Dead".
  7. Backend Overload Protection vs Push Flooding
  8. Gain (Pull): Monitoring server cannot be DDOSed by traffic spikes; it scrapes at its own controlled rate.
  9. Lose (Push): 10,000 application instances flooding the central metrics collector during a traffic spike can crash the backend.
  10. Network Topology & Firewall Traversal
  11. Gain (Push): Outbound connections (443/gRPC) traverse NAT gateways and firewalls cleanly.
  12. Lose (Pull): Requires open inbound networking paths or complex Prometheus PushProx tunnels across networks.
  13. Decoupled Architecture Simplicity
  14. Gain (Pull): Applications do not need to know the IP, hostname, or secret credentials of the monitoring server.
  15. Lose (Pull): Requires Service Discovery integrations (Kubernetes, Consul, AWS EC2 tags).

6. Interview Counter Questions

  1. Why is Prometheus fundamentally built on a Pull-Based architecture?
  2. How does Prometheus handle short-lived ephemeral batch jobs (Prometheus Pushgateway)?
  3. How does Service Discovery (Kubernetes API) enable Pull-Based metric scraping?
  4. What happens in a Push-Based architecture during a massive 10x traffic spike (Collector Overload)?
  5. What is the OpenTelemetry (OTel) Collector's role in bridging Push and Pull metric pipelines?
  6. How does Prometheus detect if an application instance has crashed (up metric)?
  7. Why is pushing metrics directly from application code to a remote database via HTTP synchronous calls an anti-pattern?
  8. How does StatsD solve push metrics using UDP local daemons?
  9. How do Thanos and Cortex scale Pull-Based Prometheus storage globally across multi-region clusters?
  10. When should you use BOTH Push and Pull in the same enterprise observability architecture?

7. Gold Standard Answers

  1. Pull allows the monitoring server to control ingestion rates, prevent backend DDOS, decouple app code from server IPs, and track target instance availability natively via failed scrapes.
  2. Batch jobs push metrics to Prometheus Pushgateway before exiting. Pushgateway retains the metrics in memory, allowing Prometheus to scrape them asynchronously during normal scrape loops.
  3. Prometheus queries the Kubernetes API Server for pods matching annotations (prometheus.io/scrape: "true"). It dynamically updates its target list, adding new pod IPs and removing dead ones.
  4. Un-buffered Push backends experience network socket exhaustion and memory crashes. Solved by running a local DaemonSet Proxy Agent (OTel Collector / StatsD) on each node to queue and batch metric pushes.
  5. OTel Collector acts as a flexible bridge: it can receive metrics via Push (OTLP gRPC) or Pull (Prometheus scraper), process/batch them, and export them via Push (Datadog/OTLP) or Pull (Prometheus endpoint).
  6. On every scrape attempt, Prometheus sets the synthetic up metric. If the target HTTP /metrics endpoint returns 200 OK, up = 1. If the connection times out or fails, up = 0 (triggering instant alerts).
  7. Synchronous HTTP metric pushes block application worker threads. Network latency or collector outages directly degrade end-to-end application response times, breaking application performance.
  8. StatsD sends metrics over fire-and-forget UDP port 8125 to a local background daemon on localhost. UDP avoids network thread blocking; the daemon aggregates metrics locally and flushes to the backend.
  9. Thanos / VictoriaMetrics attach a sidecar to Prometheus instances, uploading historical TSDB blocks to object storage (S3) and providing a unified global query engine (Thanos Querier) across clusters.
  10. Hybrid Architecture: Use Pull (Prometheus) for standard K8s microservices and infrastructure. Use Push (OTel / Datadog Agent) for serverless AWS Lambda functions, mobile apps, and edge devices.

8. Common Mistakes

  • Using Prometheus Pushgateway as a general-purpose push receiver for standard long-lived microservices (Pushgateway is strictly for batch jobs).
  • Pushing metrics synchronously over HTTP directly inside application request handlers without local UDP/agent buffering.
  • Assuming Push architectures natively detect instance crashes (Push cannot differentiate zero traffic from a crashed instance without heartbeats).
  • Failing to use local DaemonSet agents (OTel Collector) when pushing metrics from high-scale microservice fleets.
  • Opening incoming firewalls across insecure public networks to allow external Prometheus servers to pull metrics directly.

9. Keywords

Push Metrics, Pull Metrics, Prometheus, StatsD, OpenTelemetry Collector, OTLP, Prometheus Pushgateway, Service Discovery, Kubernetes Pod Annotations, up Metric, UDP 8125, Thanos, VictoriaMetrics, DaemonSet Agent, Ephemeral Jobs.

10. Cheat Sheet

  • Pull-Based (Prometheus): Server scrapes /metrics via HTTP. Best for K8s microservices. Controls load, auto-discovers targets, detects dead pods (up == 0).
  • Push-Based (StatsD / OTel): App pushes metrics outbound. Best for AWS Lambda serverless, ephemeral batch scripts, and firewalled edge devices.
  • Batch Job Solution: Use Prometheus Pushgateway to cache metrics from short-lived batch jobs for Prometheus scraping.
  • Push Best Practice: Always push to a Local Node Agent (OTel Collector / StatsD UDP) on localhost to prevent app thread blocking.
  • Crash Detection: Pull detects pod crashes immediately (up = 0); Push requires explicit heartbeat metrics to detect dead instances.
  • Scaling Pull: Use Thanos or VictoriaMetrics to aggregate multiple Prometheus pull instances into a global S3-backed dashboard.

11. Related Decisions