Common mistakes

1 min read · Chapter 9 of 10

What usually goes wrong

Symptom	Likely cause	First thing to check
Bill up 40-50% on 10% more traffic	A new high-cardinality tag, debug logs left on, indexed share creeping up, overage at on-demand rates	Usage by product, top custom metrics by tag, indexed share
Self-hosted Prometheus runs out of memory	Cardinality explosion from a label like user ID or raw URL path	Active series by label
Traces arrive in pieces	A proxy or queue drops the trace header	Whether traceparent survives each hop
Error traces missing during an incident	Head sampling dropped them	The sampling policy; tail sampling for errors
Pager fires dozens of times a day and gets ignored	Threshold alerts on noisy metrics	Alerts per real incident; SLO burn-rate alerts
Customers report the outage first	No alert, a misconfigured one, or an ignored one	The detection section of the postmortem
Data gaps with no errors anywhere	Collector buffers full, or collectors not monitored	The collectors' own metrics
Double bill or duplicate spans during migration	Vendor agent and OpenTelemetry both instrumenting the same app	Which one instruments each service
Emails or tokens found in logs	No redaction before storage	Redaction rules in the collector or pipeline
Auditor asks for last year's card-system logs	Only the hot index was kept	Retention tiers against PCI's 12 months