OpenTelemetry Collector in Production: The Configuration Nobody Writes About

Running the OpenTelemetry Collector in production is where most observability projects quietly stall. The documentation is generous about what the Collector is, the pipeline model, receivers, processors, and exporters, and almost silent about the configuration that keeps it standing when real traffic arrives. Teams deploy a Collector from a quickstart YAML, point their SDKs at it, and then spend the next three weeks debugging out-of-memory kills, dropped spans, and a backend that falls over because the Collector never learned to slow down. This post is the production configuration reference we ship at Wolf-Tech for a Collector sitting in front of Prometheus, Loki, and Tempo. It covers the exact blocks that matter, the error messages you will actually see, and the sizing guidance nobody puts in a README.

The short version: a production OpenTelemetry Collector needs four processors configured correctly before anything else matters. The memory limiter stops it from being killed. The batch processor stops it from hammering your backends. The retry and queue settings on the exporters stop data loss when a backend blips. Tail sampling stops your storage bill from tracking your traffic linearly. Get these right and the Collector becomes boring, which is exactly what you want from telemetry infrastructure.

Start With the Memory Limiter, Not the Receivers

The single most common production failure is the Collector getting OOM-killed inside its container. The default configuration has no memory ceiling, so under a traffic spike the Collector buffers more and more data until the kernel terminates it, you lose the in-flight telemetry, and the restart storm makes the incident you were trying to observe invisible. The fix is the memory_limiter processor, and it has to be the first processor in every pipeline.

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25

The check_interval controls how often the Collector measures its own heap. One second is the right default; longer intervals let a spike outrun the limiter. limit_percentage is the soft ceiling as a percentage of the container's memory limit, and spike_limit_percentage is the headroom the limiter keeps in reserve for sudden bursts. When usage crosses the soft limit, the Collector starts refusing new data and returns errors to the SDKs, which then apply their own backpressure. That is the behaviour you want: shedding load deliberately instead of dying.

Two rules make this actually work. First, set an explicit memory limit on the container (Kubernetes resources.limits.memory, or the equivalent in your orchestrator) so the percentages have something to measure against. Second, put memory_limiter first in the processors list of every pipeline. A memory limiter that runs after your batch processor is protecting the wrong thing.

The Batch Processor Is Not Optional

Without batching, the Collector forwards every span and metric point to the backend as it arrives. That means thousands of tiny writes per second to Tempo or your metrics store, which is how you turn a healthy backend into the bottleneck. The batch processor groups telemetry into reasonably sized payloads before export.

processors:
  batch:
    send_batch_size: 8192
    send_batch_max_size: 16384
    timeout: 5s

send_batch_size is the target number of items before a batch is sent, timeout forces a flush even if the batch is not full so low-traffic periods do not sit on data, and send_batch_max_size caps the batch so a burst does not produce a single oversized request that the backend rejects. The order in the pipeline is deliberate: memory_limiter first, then batch, then everything else. You want the limiter to reject data before you have spent memory assembling batches.

If you see backend rejections complaining about request size, send_batch_max_size is the knob. If you see high memory usage that correlates with export latency, the backend cannot keep up and you need to look at the queue settings below rather than making batches larger.

Retry and Queue: Where Data Loss Actually Happens

Backends go down. Tempo restarts, Loki gets rate limited, the network hiccups. What happens to your telemetry during those seconds decides whether an incident is observable. The exporter's retry_on_failure and sending_queue settings are the difference between a brief buffering and a silent hole in your traces.

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000

retry_on_failure uses exponential backoff between initial_interval and max_interval, giving up after max_elapsed_time. The sending_queue is the in-memory buffer that holds data while retries happen; queue_size is how many batches it will hold and num_consumers is how many export workers drain it in parallel. When a backend is down, the queue fills. If it fills completely, the Collector drops data and logs it, which is your signal that either the outage is longer than your queue can cover or the queue is undersized for your throughput. For telemetry you cannot afford to lose during a backend restart, the persistent queue backed by disk is worth configuring, though it trades some throughput for durability.

The failure mode to watch for is a queue that is quietly full at steady state. That means your backend is permanently slower than your ingest rate, and no retry tuning will save you. That is a capacity problem, and it is exactly the kind of thing a code quality and performance review surfaces before it becomes a 2 AM page.

Resource Detection: Enrich Once, Not in Every SDK

Every span and metric should carry host, container, and cloud metadata so you can filter by node, region, or deployment. Doing that in each application SDK means duplicated configuration and drift. The resourcedetection processor adds it centrally at the Collector.

processors:
  resourcedetection:
    detectors: [env, system, docker]
    timeout: 5s
    override: false

The detectors list runs in order and merges attributes: env reads OTEL_RESOURCE_ATTRIBUTES, system adds hostname and OS, and cloud detectors (ec2, gcp, azure) add instance and region metadata when you run there. Setting override: false means the Collector does not clobber attributes the SDK already set, so a service that deliberately reports its own identity keeps it. This is the processor that lets you answer "which node was this slow trace on" without instrumenting that question into every service.

Tail Sampling: The Config That Controls Your Bill

Head sampling, deciding to keep or drop a trace at the start, is cheap but dumb: it throws away traces before it knows whether they contained an error or a slow request. Tail sampling waits until a trace is complete and then decides, which lets you keep everything interesting and drop the healthy-path noise. This is the processor that keeps your trace storage from growing linearly with traffic.

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow
        type: latency
        latency: { threshold_ms: 500 }
      - name: healthy-sample
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

decision_wait is how long the Collector holds a trace's spans before deciding, which must be longer than your slowest realistic trace so you do not decide on an incomplete trace. num_traces is the number of in-flight traces held in memory; size it to decision_wait times your traces-per-second, with headroom. The policies are evaluated as OR: this configuration keeps every errored trace, every trace slower than 500 milliseconds, and a 5 percent sample of everything else. That combination gives you full visibility into problems while cutting storage on the healthy path by an order of magnitude.

The catch that trips teams up: tail sampling requires all spans of a trace to reach the same Collector instance. If you run multiple Collectors behind a naive load balancer, spans from one trace scatter across instances and the sampler sees incomplete traces. You need a load-balancing exporter in a two-tier setup, a first tier that routes by trace ID to a second tier that does the sampling. Skipping that step is the most common reason tail sampling "does not work" in production.

A Complete Production Pipeline

Here is how the pieces assemble into a working service block. Order inside each pipeline is not cosmetic; it is the processing sequence.

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, tail_sampling, batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp, prometheus]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [loki]
  telemetry:
    metrics:
      level: detailed
      address: 0.0.0.0:8888

Note that memory_limiter leads every pipeline and batch comes last before export, with tail_sampling placed before batch in the traces pipeline so batching operates on already-sampled data. The telemetry block exposes the Collector's own metrics on port 8888, which you should scrape. A Collector that cannot observe itself is a blind spot in the middle of your observability stack.

Reading the Error Messages

A few messages show up again and again, and their real meaning is not obvious from the text.

"data refused due to high memory usage" means the memory limiter is doing its job and shedding load. Occasional occurrences during spikes are fine; a constant stream means the container is undersized or the batch and queue settings are holding too much.

"sending queue is full" from an exporter means the backend cannot keep pace with ingest. Increasing queue_size buys time during a blip but does not fix a sustained mismatch, which is a capacity decision.

"context deadline exceeded" on export usually means backend latency has crossed the exporter's timeout. Check the backend before touching the timeout, since a longer timeout on a struggling backend just backs up the queue.

Sizing Guidance for Real Traffic

Exact numbers depend on span size and attribute cardinality, but a workable starting point: for roughly 10,000 spans per second, provision the Collector with about 1 CPU and 512 MB of memory, send_batch_size around 8192, and a sending_queue of a few thousand. At 50,000 spans per second, plan for 2 to 4 CPUs and 1 to 2 GB, and move tail sampling into its own dedicated tier so the sampling memory does not compete with ingest. Above that, scale horizontally with the trace-ID load-balancing tier described earlier rather than making a single Collector bigger. Always load test with production-shaped data, because a Collector that is comfortable with uniform synthetic spans can behave very differently under high-cardinality real traffic.

The Collector rewards being treated as the production service it is: give it a memory limit, watch its own metrics, and tune batch and queue settings against observed behaviour rather than guesses.

Where This Fits

Getting the OpenTelemetry Collector production configuration right is a small part of a larger observability practice, and it usually surfaces alongside decisions about instrumentation, backend choice, and cost. If your team is standing up observability for a growing SaaS, or inheriting a Collector config that keeps falling over, this is the kind of work we do inside custom software development and tech stack strategy engagements, and it pairs naturally with a broader code and performance review when the pipeline is a symptom of deeper capacity issues.

If you want a second pair of eyes on your Collector configuration or your observability stack, reach out at hello@wolf-tech.io or take a look at what we do at wolf-tech.io. We would rather help you make telemetry boring than watch it page you at 2 AM.