OpenTelemetry Collector Pipelines: Receivers, Processors, Exporters, and Failure Control
How to design Collector pipelines for metrics, logs, and traces without creating an unbounded queue, telemetry leak, or new single point of failure.
The OpenTelemetry Collector receives telemetry, processes it, and exports it to one or more backends. A service pipeline connects named receivers, processors, and exporters for a signal such as traces, metrics, or logs. Components listed in configuration do nothing until referenced by an enabled pipeline.
Deployment patterns
An agent runs near workloads and can collect host-local data. A gateway centralizes filtering, batching, sampling, or vendor export. Many production systems combine both. Gateways simplify policy but create capacity and availability requirements; agents increase fleet footprint. Size the pattern from event rate and failure tolerance rather than copying a diagram.
Backpressure and memory
Batch processors improve export efficiency. Memory limits, sending queues, retry policy, and persistent queue options determine behavior when a backend slows down. Unlimited retry against an outage turns telemetry into an out-of-memory incident. Define what may be dropped, for how long data should buffer, and which signals are most valuable.
Security and data governance
Use encrypted authenticated transport, least-privilege backend credentials, and explicit network policy. Attribute processors can remove or transform sensitive fields, but filtering after collection may already be too late for some data. Establish semantic conventions and a redaction policy at instrumentation time.
Operate the Collector itself
Monitor accepted, refused, dropped, queued, retried, and exported telemetry plus process memory and CPU. Version-pin the distribution because component availability and stability levels differ. Test backend failure, certificate rotation, malformed payloads, and rolling upgrades. The Collector makes telemetry routing portable; it cannot repair bad instrumentation or unlimited-cardinality labels.
Pipelines are signal-specific graphs
A receiver can feed multiple pipelines, and each pipeline orders processors before one or more exporters. Component identifiers can include names such as otlp/internal and otlp/vendor; the type selects implementation and the suffix distinguishes instances. A component merely declared under its configuration block is inactive until a service pipeline or extension references it.
Processor order changes behavior. Memory limiting should act early enough to protect the process, batching belongs after transformations that affect grouping, and sampling or filtering must occur with a clear understanding of which evidence will be lost. Logs, metrics, and traces do not support every component identically. Validate configuration with the exact Collector distribution and version rather than copying a component from the contrib catalog into a distribution that does not include it.
Agent and gateway failure domains
An agent near each workload can receive telemetry over localhost, enrich it with host context, and buffer a limited outage. A gateway centralizes tail sampling, routing, credential management, or vendor export. Combining them avoids exposing vendor credentials to every application, but creates two queues and two capacity plans. Document which layer retries, batches, samples, and drops.
Run gateways with enough replicas and topology spread, but understand exporter state. Load balancing traces randomly can break tail sampling because all spans for a trace must reach the same sampling decision. Use the supported load-balancing and routing patterns for the chosen signal. Agents deployed as DaemonSets need resource requests, update strategy, host permissions, and protection from node pressure.
Backpressure and durability
The memory limiter rejects data when configured thresholds are crossed so receivers and SDKs can apply backpressure where protocols support it. Sending queues and retries absorb transient exporter failure. Size queue capacity from measured event bytes and acceptable outage duration, not item count alone. A queue that can hold an hour of average traffic may hold only minutes during the incident burst that disabled the backend.
File-backed persistent queues can survive process restarts for supported exporters, but disk capacity, permissions, corruption, encryption, and data sensitivity become operational responsibilities. Decide which signals may be dropped and where. Metrics can often tolerate bounded gaps differently from audit logs or rare traces. Unlimited retries merely move an outage into memory or disk exhaustion.
Security and tenancy
Use TLS and authenticated OTLP endpoints across trust boundaries, separate tenant credentials and exporters, and restrict receiver exposure with network policy and firewalls. Collector configuration and environment-variable expansion can contain backend credentials; protect configuration delivery and diagnostic output. Extensions such as health checks or profiling endpoints should bind only where operators need them.
Redaction processors reduce downstream exposure but should not be the first privacy boundary. Instrumentation must avoid capturing secrets. Establish allowlists for attributes, hash only when correlation need and privacy review justify it, and test transformations against nested and malformed values. Keep telemetry from different regulatory or customer boundaries in separate pipelines or collectors when policy requires stronger isolation.
Self-observation and change control
Scrape or export the Collector’s own internal telemetry independently enough to diagnose a failed vendor exporter. Track receive and export rates, refusal and drop counts, queue occupancy, retry age, CPU, memory, garbage collection, open connections, and configuration reload or restart. Alert on divergence between accepted and exported data, not only process uptime.
Pin the Collector distribution and container digest, review component stability, and test configuration with production-like rates. Exercise backend throttling, DNS failure, expired certificates, malformed payloads, queue saturation, disk full, rolling restart, and loss of a gateway zone. A safe rollout includes rollback configuration and preserves buffered data according to the documented queue semantics.
Capacity worksheet
Measure spans, log records, and metric points per second plus representative serialized size. Multiply by burst factor, processing cost, export fan-out, retry amplification, and retention in queues. Then load-test CPU, memory, network, and backend limits. Sampling can reduce export volume but tail sampling itself requires state and memory. Attribute enrichment and regex transforms can become the dominant CPU cost.
Assign ownership for instrumentation, agent, gateway, network, credentials, and backend. A Collector SLO should reflect timely accepted-to-exported telemetry and bounded loss, with explicit exceptions during backend failure. Without that contract, teams may discover during an incident that the observability pipeline failed before the application did. Related: Prometheus Joins the CNCF as Its Second Hosted Project · Diagnosing and Fixing CrashLoopBackOff in Kubernetes
Sources: OpenTelemetry Collector, Collector configuration, deployment patterns, scaling the Collector, resiliency, Collector internal telemetry