Observability: metrics, logs and traces
07
Rules and risks
3 min read · Chapter 7 of 10
How the rules work
There's no observability regulator and no licence, so most of this section is about data rules and buyer rules that happen to land on telemetry. I'd still take them seriously, because the fines come from what's in the logs.
Personal data in logs. Telemetry carries IP addresses, user IDs, emails in URLs, auth tokens in headers and now LLM prompts. The largest GDPR fine I found about data sitting in internal systems is Meta's €91 million from Ireland's regulator in September 2024, for passwords stored in plaintext, partly for failing to notify and document the breach. France's CNIL recommends keeping access logs six months to a year, minimizing personal data in them, and storing them separately and append-only. Deleting one person's data from append-only stores is hard, so the common practice is to redact in the pipeline and keep retention short. The customer is the controller and carries the liability; the vendor is a processor.
Diagnostic files are the leak people forget. In 2023 an attacker reached support files for 134 Okta customers; some were browser recordings (HAR files) holding session tokens, and five customers' sessions were hijacked. Those files never went through a pipeline. Session replay in RUM tools has its own exposure: since a court struck down part of HHS's tracking guidance in 2024, claims over replay and tracking pixels have moved to state wiretap lawsuits against website operators.
Retention floors.
PCI DSS v4.0.1, 10.5.1
- Who it covers
- Systems in the card-data environment
- Retention
- 12 months, the last 3 immediately available
- Status (Oct 2026)
- In force; automated log review required since March 31, 2025
OMB M-26-14
- Who it covers
- US federal civilian agencies
- Retention
- 6 months actively searchable, 12 months retrievable
- Status (Oct 2026)
- In force since May 22, 2026
OMB M-21-31
- Who it covers
- US federal agencies
- Retention
- 12 months hot plus 18 cold
- Status (Oct 2026)
- Rescinded May 22, 2026
CNIL recommendation
- Who it covers
- France, under GDPR
- Retention
- 6-12 months for access logs
- Status (Oct 2026)
- Guidance
HIPAA audit controls
- Who it covers
- Covered entities and their vendors
- Retention
- No period for logs; policies kept 6 years
- Status (Oct 2026)
- Update proposed, stalled
California cybersecurity audits
- Who it covers
- Large businesses under CCPA
- Retention
- Audits cover audit-log management
- Status (Oct 2026)
- Effective January 1, 2026; first certifications 2028
The federal change is the clearest example of rules following the bill. M-26-14 cut the floor from 30 months to 12, and the memo itself says keeping "vast quantities of logging data without clear utility" was neither operationally feasible nor cost-effective. Its top maturity level grades signal: logs should generate actionable alerts covering at least 95% of baseline requirements. PCI's 12-and-3 maps neatly onto vendors' hot and archive tiers.
Buyer rules. FedRAMP decides who can sell to US federal agencies. Datadog for Government and Grafana's federal cloud say they're authorized at High, Dynatrace holds Moderate and said in July 2026 it would pursue High, and FedRAMP's new 20x process becomes mandatory on January 1, 2027. Listed US companies must file an 8-K within four business days of deciding a cyber incident is material, under SEC Item 1.05; trade groups want it rescinded, but I found no proposal as of October 2026. Telemetry is the evidence for scoping, and "we can't determine the scope" is an answer regulators read badly.
Residency and switching. Regions are fixed at setup. Datadog's sites are separate ("you cannot share data across sites"), and New Relic says the customer is "solely responsible" for choosing its region. Moving is a re-implementation. From January 12, 2027, the EU Data Act bars cloud and SaaS providers, observability vendors included, from charging EU customers switching or data-egress fees when they move.
SLAs and contracts. Most of the customer's protection sits here, and it's thin; see Liability allocation. EU financial firms have their own operational resilience rules (DORA), which I haven't covered.
What mistakes cost
- Bill shock: a tag or a debug flag can add tens of thousands of dollars a month within days, and overage history raises the next commitment.
- Ignored alerts: users find the outage first. In Splunk's UK cut of its 2025 survey, 54% said false alerts hurt morale.
- Cost-cutting blind spots: dropped logs and sampled traces lengthen the next investigation, and rehydrating the archive mid-incident costs time you don't have.
- Secrets in logs: Meta's €91 million fine was about passwords in internal systems, and Okta's 2023 breach ran through troubleshooting files.
- Logs you didn't keep: after Storm-0558, the US Cyber Safety Review Board found Microsoft had "no evidence or logs" for parts of its explanation.
- Vendor and agent failures: Datadog's 2023 outage and CrowdStrike's 2024 update, both in Liability allocation, left customers carrying most of the cost.
- The wrong region: moving an account to another region is a rebuild, with dashboards and alerts recreated by hand.