Skip to content

Metrics vs Logs vs Traces

1. Decision

Implement the Three Pillars of Observability (Metrics, Logs, Traces) using OpenTelemetry as the unified telemetry collection standard: use Metrics for real-time aggregation, monitoring, and automated alerting; use Structured Logs for detailed contextual root-cause debugging; use Distributed Tracing to visualize end-to-end request latency across microservice call graphs.

2. 30-Second Answer

Modern cloud-native observability requires combining three complementary data types: 1. Metrics (Numeric Aggregations): Numerical values aggregated over time intervals (e.g. CPU %, HTTP QPS, p99 Latency). They are lightweight, cheap to store, highly queryable (Prometheus / Datadog), and ideal for automated alerting and real-time SLA dashboards. 2. Logs (Contextual Events): Structured JSON text records of discrete events (e.g. PaymentFailed, exception stack traces). They provide full, rich contextual details required to debug specific root causes, but consume heavy storage (Elasticsearch / Loki). 3. Traces (Request Lifecycles): Directed Acyclic Graphs (DAGs) of spans representing the path of a single HTTP/gRPC request as it traverses across microservice boundaries (Jaeger / Tempo). Traces pinpoint exact bottleneck hops causing end-to-end latency spikes.

3. Decision Matrix

Option Best For Avoid When Data Volume / Cost Query Speed Primary Use Case Operational Tooling
Metrics (Numeric) Real-time monitoring dashboards, automated alerting (Prometheus / Grafana), SLO/SLA tracking, trend analysis Debugging specific individual user errors, inspecting full stack traces Very Low (Compact time-series numerical data) Ultra-Fast (<10ms Prometheus TSDB range query) Detect THAT a problem is happening (Alerting) Prometheus, VictoriaMetrics, Datadog
Logs (Event Records) Root-cause analysis, investigating specific error stack traces, audit trails, detailed event history Real-time global metric alerting (parsing terabytes of logs for alerts is slow and expensive) High (Terabytes of structured JSON data) Medium to Slow (Lucene index search e.g. Elasticsearch / Loki) Understand WHY a problem happened (Debugging) OpenSearch, Loki, Elasticsearch
Traces (Spans) Pinpointing latency bottlenecks across multi-service call chains (A \(\rightarrow\) B \(\rightarrow\) C), microservice dependency graphs Isolated single-monolith debugging Medium to High (Depends on Sampling Rate e.g. 5-10%) Fast (Span ID lookup in Jaeger / Tempo) Identify WHERE in the system a problem happened (Localization) OpenTelemetry, Jaeger, Grafana Tempo

4. Selection Rules

Requirement Best Choice Rationale / Example
Automated PagerDuty Alerting on Service Degradation Metrics (Prometheus) Alert if sum(rate(http_requests_total{status=~"5.."}[5m])) > 10
Investigate Specific User Payment Failure Stack Trace Structured Logs (Loki/ES) Query user_id="usr_123" AND event="PaymentFailed" to read full exception
Identify Which Microservice Is Causing 3s Latency Spikes Distributed Traces (Tempo) Trace waterfall graph highlights 2,800ms bottleneck in Inventory-Service span
Unified Vendor-Neutral Telemetry SDK Instrumenting Code OpenTelemetry (OTel) Single OTel SDK exports Metrics, Logs, and Traces to any backend destination

5. Tradeoffs

  1. Numeric Storage Efficiency (Metrics) vs Contextual Detail (Logs)
  2. Gain (Metrics): Tiny storage footprint; retain 1 year of QPS metrics for pennies.
  3. Lose (Metrics): Zero contextual detail (cannot tell which user ID failed).
  4. End-to-End Latency Visibility (Traces) vs Network & Storage Overhead
  5. Gain (Traces): Visualizes request propagation paths across 50 microservices automatically.
  6. Lose (Traces): 100% tracing ingestion consumes massive network bandwidth and disk space (requires Sampling).
  7. Structured JSON Logs vs Storage Inflation
  8. Gain (Logs): Searchable key-value attributes (tenant_id, db_latency_ms) enable precise querying.
  9. Lose (Logs): Ingesting terabytes of un-sampled JSON logs per day incurs massive Datadog/Elasticsearch bills.
  10. OpenTelemetry Standardization vs Proprietary Agent Simplicity
  11. Gain (OTel): 100% vendor-agnostic; switch from Datadog to Grafana Tempo without changing app code.
  12. Lose (OTel): Configuring OTel Collector pipelines and exporters requires initial DevOps setup.
  13. Trace Context Propagation Overhead
  14. Gain: Links logs, metrics, and traces together using a unified trace_id.
  15. Lose: App code and HTTP/gRPC clients MUST pass traceparent headers down the network call tree.

6. Interview Counter Questions

  1. What are the "Three Pillars of Observability" and how do they work together to resolve an incident?
  2. How does trace_id correlation link Logs, Metrics, and Traces together in Grafana?
  3. What is OpenTelemetry (OTel) and why has it become the industry standard over proprietary vendor SDKs?
  4. What is the difference between Tail-Based Sampling and Head-Based Sampling in Distributed Tracing?
  5. Why are Metrics better than Logs for real-time alerting?
  6. How does Prometheus store time-series metrics efficiently (TSDB chunk compression)?
  7. What is the RED Method (Rate, Errors, Duration) vs USE Method (Utilization, Saturation, Errors)?
  8. How do you control logging costs in high-volume microservice environments?
  9. What is the OTel Collector and what are its 3 pipeline components (Receivers, Processors, Exporters)?
  10. How does eBPF-based auto-instrumentation (Cilium Hubble / Pixie) complement manual OpenTelemetry code instrumentation?

7. Gold Standard Answers

  1. Metrics detect THAT a problem exists (Alert fires: p99 latency \(>2\text{s}\)). Traces localize WHERE the bottleneck is (Payment-Service RPC hop). Logs explain WHY it failed (PostgreSQL connection pool exhausted).
  2. Embed the current trace_id inside structured JSON log outputs ({"trace_id": "abc123", "msg": "failed"}). In Grafana, clicking a metric spike opens the Trace graph, and clicking a trace span filters Logs matching that trace_id.
  3. OpenTelemetry (CNCF standard) provides vendor-neutral APIs, SDKs, and Collector pipelines. It allows developers to instrument code once and export telemetry to any backend (Prometheus, Jaeger, Datadog, New Relic).
  4. Head-Based: Decides to sample a trace at request start (e.g. random 5% sample). Misses rare 500 errors! Tail-Based: Buffers 100% of spans in memory; samples and persists 100% of traces containing errors or high latency.
  5. Metrics are pre-aggregated numbers (tiny storage footprint). Querying 100,000 requests/sec via Prometheus takes \(<10\text{ms}\). Searching terabytes of text logs to calculate error rates takes minutes and thrashes disk I/O.
  6. Prometheus TSDB uses Gorilla compression (XOR delta-of-delta timestamp and value compression), packing time-series metrics into ~1.37 bytes per sample in memory/disk.
  7. RED Method (Request-focused): Rate (QPS), Errors (5xx rate), Duration (Latency SLAs). USE Method (Resource-focused): Utilization (% busy), Saturation (Queue depth), Errors (Device errors).
  8. Implement: (1) Log Level tuning (WARN/ERROR in prod; DEBUG off), (2) Dynamic Sampling, (3) Filtering verbose health checks at OTel Collector, (4) Indexing only critical keys in Grafana Loki.
  9. Receivers: Ingest data (OTLP, Prometheus format). Processors: Batch, filter, memory-limit, or sample telemetry. Exporters: Send telemetry to backends (Jaeger, Prometheus, Datadog).
  10. eBPF automatically intercepts network calls and HTTP requests at the Linux kernel layer with ZERO code modification or SDKs. OTel provides deep in-code business context (user IDs, domain attributes).

8. Common Mistakes

  • Using raw un-sampled logs to calculate high-velocity QPS metrics, causing massive log ingestion bills.
  • Instrumenting code using proprietary vendor SDKs (Datadog/NewRelic SDKs), causing permanent vendor lock-in.
  • Collecting 100% Head-Based trace samples in high-traffic applications, blowing out network bandwidth and storage costs.
  • Outputting un-structured plain-text log strings instead of structured JSON logs ({"level": "ERROR", "msg": ...}).
  • Failing to correlate logs with traces by omitting trace_id fields in application logging configurations.

9. Keywords

Observability, Three Pillars, Metrics, Logs, Traces, OpenTelemetry (OTel), OTLP, OTel Collector, Jaeger, Prometheus, Grafana Tempo, Loki, Trace ID, Spans, Head-Based Sampling, Tail-Based Sampling, RED Method, USE Method, TSDB, eBPF.

10. Cheat Sheet

  • Three Pillars Roles:
  • Metrics \(\rightarrow\) Detect THAT a problem exists (Prometheus Alerting).
  • Traces \(\rightarrow\) Identify WHERE in the call chain the bottleneck lives (Jaeger/Tempo).
  • Logs \(\rightarrow\) Explain WHY the error happened (OpenSearch/Loki JSON stack traces).
  • Correlation: Always inject trace_id into structured JSON logs to link Traces directly to Logs in Grafana.
  • Instrumentation Standard: Use OpenTelemetry (OTel) for vendor-agnostic code instrumentation and telemetry pipelines.
  • Tracing Sampling: Use Tail-Based Sampling in OTel Collector to capture 100% of error/high-latency traces while dropping 95% of normal traces.
  • Monitoring Frameworks: Use RED Method (Rate, Errors, Duration) for APIs; use USE Method (Utilization, Saturation, Errors) for Infrastructure.
  • Cost Control: Store Metrics for long-term trends; sample Logs and Traces to control cloud storage bills.

11. Related Decisions