Observability Integration Pitfalls: What Goes Wrong When You Wire Prometheus, Grafana, OTel, and Sentry Together

Wiring Prometheus, Grafana, OpenTelemetry, and Sentry into one stack looks like a solved problem. Each tool has excellent docs, each has a quickstart, and each works fine on its own. The observability integration pitfalls appear in the seams: the places where one tool's defaults collide with another's assumptions. A metric label that Prometheus happily accepts blows up your storage. A batch processor setting that works in staging silently drops spans in production. Two sampling decisions made by two SDKs produce traces with no parents.

We have written a step-by-step integration checklist for building this stack from scratch. This post is the other half of the story: the six failure modes we see most often when auditing observability setups, and the specific configuration that fixes each one.

The Six Observability Integration Pitfalls, Ranked by Damage

Ordered by how expensive they are to discover in production:

1. Prometheus metric cardinality explosion from careless labels
2. OTel Collector dropping spans under backpressure
3. Grafana clock skew causing phantom gaps in dashboards
4. Sentry and OTel sampling conflicts producing orphaned traces
5. Alert fatigue from duplicated alerting rules
6. The performance cost of running all four agents on one server

Pitfall 1: Cardinality Explosion From Naive Labels

Prometheus stores one time series per unique combination of metric name and label values. Add a user_id label to a request counter and you no longer have one series, you have one series per user. Add path without normalisation and every URL with an ID in it becomes its own series. Memory usage grows linearly with active series, and a Prometheus instance that ran comfortably at 200k series will start OOM-killing itself at 5 million.

The insidious part is the delay. Cardinality grows with traffic and feature launches, not with deploys, so the explosion lands weeks after the label was added and nobody connects the two events.

The fix. Never use unbounded values (user IDs, session IDs, raw URLs, container hashes) as label values. Normalise routes to templates (/api/orders/{id}), and enforce the rule at scrape time with metric_relabel_configs so one bad exporter cannot take down the server:

metric_relabel_configs:
  - source_labels: [user_id]
    regex: '.+'
    action: labeldrop

Watch prometheus_tsdb_head_series and alert when it grows more than 20 percent week over week. That single alert has paid for itself in every audit we have run.

Pitfall 2: The OTel Collector Dropping Spans Under Backpressure

This is the pitfall people search for after the fact, usually phrased as "opentelemetry collector missing spans". The default batch processor configuration is tuned for demos. Under real load, the Collector accepts spans faster than it exports them, queues fill, and spans are dropped with nothing more than a counter increment to show for it.

The two settings that matter are send_batch_size and timeout on the batch processor, combined with the exporter queue:

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  batch:
    send_batch_size: 8192
    send_batch_max_size: 10000
    timeout: 5s

exporters:
  otlp:
    endpoint: tempo:4317
    sending_queue:
      enabled: true
      queue_size: 5000
    retry_on_failure:
      enabled: true
      max_elapsed_time: 300s

A send_batch_size that is too small (the default is 8192, but many tutorials set it to 512) means more export calls per second and earlier backpressure. A timeout that is too aggressive flushes half-empty batches. The memory_limiter must come first in the processor chain, and the exporter queue is what absorbs a backend blip without data loss.

You will know this pitfall by the metric otelcol_processor_dropped_spans being nonzero. Alert on it. For the full production reference, see our OTel Collector configuration guide.

Pitfall 3: Grafana Clock Skew and Phantom Dashboard Gaps

A dashboard shows a two-minute gap in metrics. The on-call engineer assumes an outage, digs through logs, finds nothing. The real cause: the Grafana host, the Prometheus host, and the application host disagree about what time it is by 40 seconds.

Prometheus timestamps samples at scrape time using its own clock. Grafana renders using its own clock and the browser's. If any of these drift, recent data appears to be missing because Grafana is asking for samples "newer" than what Prometheus believes exists yet. The gaps always appear at the right edge of the dashboard and always resolve themselves on refresh, which is exactly the behaviour that erodes trust in the whole stack.

The fix has three parts. First, run chrony or systemd-timesyncd on every host and alert on drift: node_timex_offset_seconds above 0.05 is a problem. Second, set a sensible minimum interval in the Grafana data source (Scrape interval in the Prometheus datasource settings should match your actual scrape interval, typically 15s or 30s) so Grafana does not interpolate points that cannot exist. Third, in panels that aggregate over time, use $__rate_interval instead of hard-coded ranges, which keeps queries aligned with scrape resolution.

Pitfall 4: Sentry and OTel Sampling Conflicts That Orphan Traces

Sentry's SDKs implement their own tracing with their own sampler. OpenTelemetry has its sampler. Run both in one application without coordinating them and each makes an independent keep-or-drop decision. The result is traces where Sentry kept the child span but OTel dropped the parent: orphaned traces that render as fragments, with the trace waterfall missing its root.

The symptom is distinctive: Tempo or Jaeger shows traces that begin mid-request, and Sentry performance data disagrees with OTel data about traffic volume for the same endpoint.

The fix is to let exactly one system make the sampling decision and have the other respect it. Since Sentry supports OpenTelemetry directly, the clean setup routes Sentry through OTel instead of running two tracers:

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  skipOpenTelemetrySetup: true,
  tracesSampleRate: undefined,
});

Then configure sampling once, in the OTel SDK (parentbased_traceidratio keeps decisions consistent across services), and attach Sentry's span processor to the OTel tracer provider. In PHP and Python setups where that wiring is not available, the fallback is to set both samplers to the same rate and use a traces_sampler callback that honours the incoming parent decision. The rule either way: one sampler decides, everyone else inherits.

Pitfall 5: Duplicated Alerting Across Alertmanager and Sentry

Prometheus Alertmanager alerts on symptoms: error rate above 2 percent, p95 latency above 800ms. Sentry alerts on issues: new exception type, regression after a release. Configure both without a plan and the same incident pages you twice, through two channels, with two different links. After the third duplicated 2 AM page, engineers start muting channels, and then a real alert dies in a muted channel.

The fix is an explicit division of responsibility, written down:

• Alertmanager owns symptom alerts. Anything rate-based, latency-based, or saturation-based. These are the pages.
• Sentry owns regression alerts. New issue types, issues that reappear after being resolved, release-health thresholds. These go to a triage channel, not a pager.

Then enforce it: delete Sentry alert rules that fire on volume (Alertmanager already covers that), and use Alertmanager inhibition rules so a firing symptom alert suppresses its downstream noise. The test for a healthy setup: any single incident produces exactly one page and at most one triage notification.

Pitfall 6: Four Agents on One Server

On a single-server deployment, the observability stack itself becomes a tenant. Prometheus, the OTel Collector, Grafana, and a Sentry relay together consume 1.5 to 2.5 GB of RAM and a steady slice of CPU before your application serves a single request. We have audited setups where observability consumed 30 percent of the machine and the team was evaluating a server upgrade to make room for it.

The fix is deliberate budgeting. Cap each component (--storage.tsdb.retention.time=15d and --storage.tsdb.retention.size=8GB for Prometheus, the memory_limiter from Pitfall 2 for the Collector, container memory limits for Grafana). Widen scrape intervals to 30s or 60s where 15s buys nothing. Sample traces at 10 percent instead of 100. If the budget still does not fit, hosted Grafana Cloud or Sentry SaaS for the heavy components is usually cheaper than the next server tier.

The Pattern Behind All Six

Every one of these pitfalls follows the same shape: each tool behaves correctly by its own defaults, and the failure lives in the seam between them. That is why observability problems resist debugging tool by tool, and why the fix is always a decision about the seam: who samples, who alerts, who owns the label budget, whose clock wins.

If your stack shows any of these symptoms, an outside review is often faster than another internal debugging round. Wolf-Tech runs performance and code quality audits that cover observability configuration, and helps teams make tech stack decisions like self-hosted versus SaaS for the monitoring layer. Write to hello@wolf-tech.io or visit wolf-tech.io if you want a second pair of eyes on your setup.