Observability in Cloud-Native Systems: Metrics, Logs, and Traces

“Monitoring” tells you whether something is wrong; observability is about being able to ask arbitrary new questions about a system’s internal state without shipping new code to answer them. In cloud-native systems — where a single request might cross a dozen services — that distinction matters enormously, and it’s usually built from three complementary signal types: metrics, logs, and traces.

Metrics: cheap, aggregated, great for alerting

A metric is a numeric measurement recorded over time — request count, latency, memory usage — typically pre-aggregated (a counter, a gauge, a histogram) rather than stored as individual raw events. Prometheus is the de facto standard in cloud-native environments, scraping metrics endpoints on a fixed interval:

http_requests_total{method="GET", status="200"} 84213
http_request_duration_seconds_bucket{le="0.5"} 79231

histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

Metrics are cheap to store and query at scale precisely because they’re aggregated — you lose the ability to inspect any individual request, but gain the ability to cheaply track trends and alert on thresholds across millions of events.

Logs: detailed, per-event, expensive at scale

A log is a discrete, timestamped record of a specific event, typically with unstructured or semi-structured detail — an error message, a stack trace, a specific request’s parameters. Structured logging (emitting JSON rather than free-text) is what makes logs actually queryable at scale rather than requiring regex archaeology:

{"timestamp": "2026-04-08T14:23:01Z", "level": "error", "service": "checkout", "trace_id": "abc123", "msg": "payment gateway timeout", "user_id": "u_9182"}

{service="checkout"} |= "error" | json | trace_id="abc123"

Logs answer “what exactly happened” for a specific event in a way aggregated metrics fundamentally can’t, at the cost of considerably higher storage and query cost at high volume — which is why log sampling and retention policies matter far more in practice than they do for metrics.

Traces: following one request across service boundaries

A trace follows a single request as it flows through multiple services, composed of spans — one per unit of work (an HTTP call, a database query) — each carrying a start time, duration, and parent-child relationship to other spans in the same trace:

Trace: abc123
├─ span: api-gateway (12ms)
│  └─ span: checkout-service (340ms)
│     ├─ span: inventory-check (45ms)
│     └─ span: payment-gateway-call (280ms)  ← the slow one

# OpenTelemetry instrumentation, conceptually
tracer.start_span("payment-gateway-call")

This is the signal that answers “which specific service in this request’s path is actually slow” — a question metrics alone (which are per-service, not per-request) and logs alone (which lack the causal parent-child linkage across services) can’t answer nearly as directly.

OpenTelemetry: one instrumentation standard for all three

OpenTelemetry (OTel) has become the standard vendor-neutral instrumentation layer for emitting metrics, logs, and traces from application code, decoupling “how do I instrument my code” from “which backend do I send this to” — the same application instrumentation can export to Prometheus, Jaeger, Datadog, or any other OTel-compatible backend without changing the instrumentation itself:

receivers:
  otlp:
    protocols:
      grpc:
      http:
exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
  otlp/jaeger:
    endpoint: "jaeger:4317"

An OpenTelemetry Collector typically sits between instrumented applications and backend storage, receiving all three signal types over a common protocol (OTLP) and routing/processing/exporting them onward.

Why you need all three, not just the “best” one

A common instinct is to ask which signal is most valuable and invest there exclusively — but each answers a genuinely different question, and incidents routinely require moving between all three: metrics first surface that latency spiked cluster-wide (an alert fires); traces narrow down which specific service in the request path is responsible; logs from that specific service, filtered by the trace ID the trace provided, reveal the actual error causing the slowdown. Correlating all three via a shared trace_id (propagated through logs and linked from the trace view) is what turns “three separate tools” into one coherent investigation workflow rather than three disconnected data sources.

1. Alert: p99 latency > 2s (metric)
2. Trace shows payment-gateway-call span consistently slow (trace)
3. Logs for that service, filtered by trace_id, show connection pool exhaustion (log)

The cost dimension: sampling and retention

At high request volume, capturing every trace and every log line in full detail becomes prohibitively expensive — sampling (recording only a percentage of traces, or all traces above a latency/error threshold) and tiered retention (recent data at full fidelity, older data aggregated or discarded) are standard practical necessities, not shortcuts. A well-designed observability strategy samples deliberately rather than either drowning in cost or silently missing the specific slow/failing requests that matter most.

The underlying principle

The “three pillars” framing is really about matching the right signal to the right question: aggregated trends and alerting (metrics), the exact detail of one specific event (logs), and causal, cross-service request flow (traces). A system instrumented with only one of the three can tell you that something’s wrong or what happened in isolation, but reliably answering why, across a distributed, multi-service request path, is what actually needs all three working together.