Observability in Cloud-Native Systems: Metrics, Logs, and Traces

“Monitoring” tells you whether something is wrong; observability is about being able to ask arbitrary new questions about a system’s internal state without shipping new code to answer them. In cloud-native systems — where a single request might cross a dozen services — that distinction matters enormously, and it’s usually built from three complementary signal types: metrics, logs, and traces.

Metrics: cheap, aggregated, great for alerting

A metric is a numeric measurement recorded over time — request count, latency, memory usage — typically pre-aggregated (a counter, a gauge, a histogram) rather than stored as individual raw events. Prometheus is the de facto standard in cloud-native environments, scraping metrics endpoints on a fixed interval:

http_requests_total{method="GET", status="200"} 84213
http_request_duration_seconds_bucket{le="0.5"} 79231

histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

Metrics are cheap to store and query at scale precisely because they’re aggregated — you lose the ability to inspect any individual request, but gain the ability to cheaply track trends and alert on thresholds across millions of events.

Logs: detailed, per-event, expensive at scale

A log is a discrete, timestamped record of a specific event, typically with unstructured or semi-structured detail — an error message, a stack trace, a specific request’s parameters. Structured logging (emitting JSON rather than free-text) is what makes logs actually queryable at scale rather than requiring regex archaeology:

{"timestamp": "2026-04-08T14:23:01Z", "level": "error", "service": "checkout", "trace_id": "abc123", "msg": "payment gateway timeout", "user_id": "u_9182"}

{service="checkout"} |= "error" | json | trace_id="abc123"

Logs answer “what exactly happened” for a specific event in a way aggregated metrics fundamentally can’t, at the cost of considerably higher storage and query cost at high volume — which is why log sampling and retention policies matter far more in practice than they do for metrics.

Traces: following one request across service boundaries

A trace follows a single request as it flows through multiple services, composed of spans — one per unit of work (an HTTP call, a database query) — each carrying a start time, duration, and parent-child relationship to other spans in the same trace:

Trace: abc123
├─ span: api-gateway (12ms)
│  └─ span: checkout-service (340ms)
│     ├─ span: inventory-check (45ms)
│     └─ span: payment-gateway-call (280ms)  ← the slow one

# OpenTelemetry instrumentation, conceptually
tracer.start_span("payment-gateway-call")

This is the signal that answers “which specific service in this request’s path is actually slow” — a question metrics alone (which are per-service, not per-request) and logs alone (which lack the causal parent-child linkage across services) can’t answer nearly as directly.

OpenTelemetry: one instrumentation standard for all three

OpenTelemetry (OTel) has become the standard vendor-neutral instrumentation layer for emitting metrics, logs, and traces from application code, decoupling “how do I instrument my code” from “which backend do I send this to” — the same application instrumentation can export to Prometheus, Jaeger, Datadog, or any other OTel-compatible backend without changing the instrumentation itself:

receivers:
  otlp:
    protocols:
      grpc:
      http:
exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
  otlp/jaeger:
    endpoint: "jaeger:4317"

An OpenTelemetry Collector typically sits between instrumented applications and backend storage, receiving all three signal types over a common protocol (OTLP) and routing/processing/exporting them onward.

Why you need all three, not just the “best” one

A common instinct is to ask which signal is most valuable and invest there exclusively — but each answers a genuinely different question, and incidents routinely require moving between all three: metrics first surface that latency spiked cluster-wide (an alert fires); traces narrow down which specific service in the request path is responsible; logs from that specific service, filtered by the trace ID the trace provided, reveal the actual error causing the slowdown. Correlating all three via a shared trace_id (propagated through logs and linked from the trace view) is what turns “three separate tools” into one coherent investigation workflow rather than three disconnected data sources.

1. Alert: p99 latency > 2s (metric)
2. Trace shows payment-gateway-call span consistently slow (trace)
3. Logs for that service, filtered by trace_id, show connection pool exhaustion (log)

The cost dimension: sampling and retention

At high request volume, capturing every trace and every log line in full detail becomes prohibitively expensive — sampling (recording only a percentage of traces, or all traces above a latency/error threshold) and tiered retention (recent data at full fidelity, older data aggregated or discarded) are standard practical necessities, not shortcuts. A well-designed observability strategy samples deliberately rather than either drowning in cost or silently missing the specific slow/failing requests that matter most.

The underlying principle

The “three pillars” framing is really about matching the right signal to the right question: aggregated trends and alerting (metrics), the exact detail of one specific event (logs), and causal, cross-service request flow (traces). A system instrumented with only one of the three can tell you that something’s wrong or what happened in isolation, but reliably answering why, across a distributed, multi-service request path, is what actually needs all three working together.

Cardinality and data contracts

Every attribute is a cost and privacy decision. Unbounded user IDs, request URLs, container IDs, or exception text can create metric series explosions and expensive indexes. Keep metric dimensions bounded, move high-cardinality investigation keys to logs or traces, and define semantic conventions shared by producers and collectors. Schema changes need compatibility and ownership just like an API.

Do not log secrets, authentication headers, full request bodies, or unnecessary personal data. Redact as close to instrumentation as possible because downstream filtering means sensitive data has already crossed a boundary. Set retention and access by signal and purpose; “keep everything” is neither an observability strategy nor a defensible cost control.

Alert from user impact

Page on symptoms that require timely human action: error-budget burn, sustained latency, failed critical work, or exhausted capacity. Dashboard every available metric if useful, but avoid paging on transient CPU or individual container restarts without demonstrated user impact. Multi-window burn-rate alerts balance fast incident detection with resistance to short noise.

For each alert, link service ownership, SLO, dashboard, recent changes, and a tested runbook. Measure whether alerts are actionable and retire those that operators routinely ignore. Observability succeeds when it shortens detection and diagnosis while preserving evidence, not when telemetry volume grows.

Related:

Sources: