Key concepts

In telemetry the unit of identity is the resource: the service name, its version, the environment, the host, the container, the Kubernetes pod, the cloud account and region. OpenTelemetry standardizes these attribute names, and a trace adds a trace ID and span IDs that follow a request across services.

Vendors bill on entities too, and they don't agree on which entity counts:

Containers, serverless functions and autoscaling make "how many hosts" a moving number, so vendors bill on percentiles of hourly counts or "host-equivalents". The identity most often missing is ownership. Which team owns checkout-api? That lives in a service catalog if anywhere, and an alert without an owner pages whoever happens to be on call.

"Agent" has a double meaning, often on one product page: the software on each host that collects telemetry, and an AI agent. I'd write "host agent" and "AI agent" every time. Telemetry also carries other people's identities, such as user IDs, IP addresses and emails in URLs, which is where the privacy rules come in.

A piece of telemetry moves through states: emitted, collected, processed, ingested, indexed or stored, tiered down, deleted. Each change of state is a pricing event, and regulators have started naming the states: PCI's card-data rules require logs to be "immediately available" for three months, and the US federal memo M-26-14 distinguishes "actively searchable" from "retrievable".

Rehydration is fading. Since 2025, Datadog's Archive Search, Elastic's frozen tier and lakehouse engines let you query old data where it sits.

An alert goes from inactive to pending to firing, then acknowledged and resolved. An incident is detected, triaged, mitigated, resolved, reviewed and closed. Standards have states too: OpenTelemetry's traces, metrics and logs are stable, profiles reached alpha in March 2026, and the conventions for AI and LLM telemetry are still marked "Development".

The contract has states as well: free tier, annual commitment, monthly usage, overage, true-up, renewal. Seat reductions at Datadog take effect only at renewal, and overage history tends to become next year's commitment, so commitments ratchet up.

There's no single system of record.

The billing ledger is the one I'd watch. The vendor's meter decides the bill, and customers rarely have an independent count. Datadog bills custom metrics on the monthly average of hourly counts, and hosts on the 99th percentile of hourly counts, so the top 1% of hours is forgiven and a week-long spike is billed. A pipeline gives the customer its own meter of what was sent, which helps in a billing dispute.

For security and compliance teams, logs are the record: PCI requires a year of audit logs, SOC 2 auditors want evidence, and a breach investigation depends on what was kept. For the reliability side the same logs are disposable signal. The two retention philosophies collide whenever one platform serves both teams.

Most rules in observability are ones the customer writes: what to sample, redact and index, for how long, when to alert and who gets paged. They live in collector and alerting configuration, and they decide both the bill and what an investigator will find.

SLOs matter most for signal. Google's SRE Workbook recommends paging on burn rate over two windows, so a short spike pages nobody and a real outage pages fast:

That's for a 99.9% target over 30 days. At a burn rate of 1,000, the whole month's budget is gone in about 43 minutes. The math breaks for low-traffic services: at ten requests an hour, one failure spends about 14% of the monthly budget, so teams add synthetic traffic, group small services or lower the target. An error-budget policy closes the loop: while the SLO is met, releases continue, and when the budget is spent, they pause.

Google's on-call guidance sets a bar few teams meet: every page actionable, at most two incidents in a 12-hour shift, and on-call capped at a quarter of an engineer's time. Vendors add their own rules (cardinality limits, rate limits, index quotas), which are product decisions about how hard to stop a customer's mistake.

Everything in observability is measured over a window. Alerts evaluate over minutes, SLOs over a rolling 28 or 30 days, burn rates over 5 minutes to 3 days, and billing over hours, months and contract years. Retention clocks usually start when data arrives, so late mobile or batch data can land in the wrong window.

The meaning of a field has a date. OpenTelemetry's semantic conventions are versioned, and renames break things quietly: database spans renamed db.system to db.system.name in version 1.33.0 in 2025, and dashboards keyed on the old name stopped matching.

Dates I'd keep on a calendar as of October 2026:

Take the dispatch company's commitment renewing in March. Overage from a cardinality spike in January sits in the usage history the vendor brings to the renewal, so a mistake fixed in a week can still raise next year's commitment.

The collection side is now mostly standard:

• OTLP, OpenTelemetry's protocol for sending traces, metrics and logs, is stable and accepted by every major vendor.
• W3C Trace Context is the traceparent header that carries a trace ID between services.
• Semantic conventions name attributes the same way everywhere (HTTP stable since 2023, databases since 2025, AI still in development).
• Prometheus exposition format and PromQL are the de facto standard for metrics and the closest thing to a common query language; many backends accept PromQL.

The storage and query side has no standard. Logs and traces are queried in each vendor's language: LogQL and TraceQL at Grafana, SPL at Splunk, KQL at Microsoft, ES|QL at Elastic, DQL at Dynatrace, Datadog's own syntax, SQL on ClickHouse. Dashboards and alert rules are written in these languages, which is why they don't port.

Every major vendor now ships its own OpenTelemetry distribution, and the distributions recreate some lock-in. Splunk gives official support only for its own and "best-effort" support upstream. Datadog's wasn't FedRAMP or FIPS compliant when the research was done, which steers government buyers back to the proprietary agent. Tail sampling is "often tied to a specific vendor", in OpenTelemetry's own words. Datadog's FY2025 annual report doesn't mention OpenTelemetry at all; it sells "a single agent" with over 1,000 integrations. Its price list charges $0.50 per GB for spans ingested over OTLP against $0.10 for its own APM ingestion, and application metrics sent through OpenTelemetry generally count as custom metrics.

Vendors are also giving their collection technology away, which is what you do once collection stops being where you make money: Elastic donated its eBPF profiler and Grafana its Beyla. For AI, two newer interfaces matter. MCP lets AI agents query observability tools, and Datadog said MCP tool calls grew more than 22-fold from late 2025 to mid-2026 (company figure). GitHub issues and pull requests are the hand-off between observability AI and coding agents.

Concentration runs both ways. Datadog's largest customer, widely reported to be OpenAI, renewed at nine figures in 2026 but is cutting usage from July, and Datadog built that into its guidance. Coinbase's roughly $65 million a year Datadog bill was restructured in 2023 after Coinbase built a parallel open-source stack. The biggest customers can credibly leave, and the threat works best at renewal.

The pipeline is the new switching point. Whoever owns it decides which backend gets which data, which is why security and observability vendors have both bought pipelines since 2025.

I'd write down early which single failure blinds us during an incident. For many teams it's the collector tier: in OpenTelemetry's 2026 survey of collector users, 23% didn't monitor their collectors at all.

Observability has no licensing regime. Nobody needs permission to collect or store telemetry, and no regulator supervises observability vendors as such. Rules reach this industry through four other doors.

GDPR and CCPA don't mention logs; they apply because logs carry IP addresses, user IDs, emails and tokens. SOC 2 is a contract artifact with no legal force, and in practice it's the report every vendor is asked for first.

The test I'd use on any requirement: is it about the data, the buyer or the vendor? Data rules follow the data into every tool, buyer rules only matter if you sell to that buyer, and vendor rules mostly live in contracts.

Rules reverse too. The US federal logging memo M-21-31 was rescinded on May 22, 2026, a federal court struck down part of HHS's guidance on tracking technologies in 2024, and the HIPAA Security Rule update has stalled. Product reversals follow public pressure: after the Storm-0558 intrusion in 2023, which agencies could only detect with logs from a premium tier, Microsoft gave standard customers more than 30 log types at no extra cost.

The customer carries most of the risk in observability, and the contracts say so.

The SLAs are short. Grafana Cloud promises 99.5% of requests succeed each month and credits 10% to 100% of the affected service's monthly fee as availability drops; it also covers alert evaluation, so a missed alert is contractually defined there. New Relic targets 99.8%, with termination as the remedy if it misses badly two months running, and I found no public Datadog SLA to compare.

Datadog's own outage on March 8, 2023 shows the vendor side. A security update auto-applied across its nodes knocked out networking, and the company said it lost about $5 million of revenue, because under usage pricing it couldn't bill for data it couldn't ingest. Customers were blind for most of a day and got credits at best.

CrowdStrike's July 2024 outage is the extreme case for any privileged host agent, which observability agents are becoming as they take on security work. One estimate put direct losses to Fortune 500 companies at $5.4 billion, and CrowdStrike says its liability cap limits its exposure to "single-digit millions". A court let Delta's negligence claims proceed in May 2025.