How it works

How telemetry becomes a fix, step by step

Here's the slow job assignment from the moment the code records it to the fix. Much of the data is sampled out or dropped before it's kept, usually to control cost. And an alert only helps if someone acts on it; noisy alerts get ignored, and that's where many incidents slip through.

1. Emitted: the scheduling service records a latency metric, a log line and a span. Spans come from an SDK the team calls in code, from auto-instrumentation that hooks into common frameworks, or from eBPF, which watches network calls from the Linux kernel without code changes. Real-user monitoring (RUM) is emitted from users' browsers and phones. What can go wrong: a proxy or message queue drops the trace header and the trace splits in two; a host with a bad clock produces negative durations.
2. Kept: a collector on each host batches the data and adds host and Kubernetes details, then a pipeline filters, samples, redacts and routes it. Head sampling decides cheaply at the start of a trace. Tail sampling waits until the trace is complete and keeps the errors and slow ones, but it holds whole traces in memory, and OpenTelemetry's docs say it can take "dozens or even hundreds" of servers at scale. The platform indexes some data (fast to search, expensive) and stores the rest cheaply. What can go wrong: head sampling at 1% also drops 99% of error traces; scaling a tail-sampling tier can split traces silently; a collector buffer fills while the backend is slow, and data is lost during the very incident you need it for.
3. Alerted: a rule fires on a threshold, an anomaly or an SLO (service level objective, an internal target such as 99.9% of requests succeeding over 30 days). SLO alerts fire on burn rate, how fast the error budget (the 0.1% you're allowed to fail) is being spent. What can go wrong: Google's SRE Workbook shows that a plain alert on a 0.1% error rate over ten minutes could fire up to 144 times a day while the service still meets its SLO, which trains people to ignore the pager.
4. Investigated: someone looks, usually going from a dashboard to a trace to the logs, then to recent deploys and feature-flag changes. Google budgets about six hours of engineering per incident, postmortem included. This is the step AI SRE products target. What can go wrong: the trace that would show the cause was sampled out; the logs must be pulled back from the archive; the observability vendor itself is down.
5. Resolved: the team mitigates (roll back, fail over, turn off a flag), fixes properly later and writes a blameless postmortem with action items. What can go wrong: the action items never close, and the incident comes back a quarter later.

Two exits sit off that path. Dropped is data sampled out or filtered because keeping it was too costly; it can't be recovered. Ignored is an alert nobody acted on, usually because the same rule had fired falsely many times. In Splunk's 2025 survey, 52% of respondents said they struggle with high volumes of false alerts.

The cardinality column is worth remembering. A customer ID on a metric multiplies the series by the number of customers, while on a span it adds almost nothing, so per-customer questions belong in traces.

Collecting the data

Most large companies end up with a mix: in Grafana's 2026 survey, 65% of respondents were investing in both Prometheus and OpenTelemetry.