Rules and risks

How the rules work

Most AI products face no rule about how to test them. Testing duties exist for EU high-risk systems and the most capable general-purpose models, medical devices, bank models (generative AI now excluded in the US) and through contracts; everything else is general law applied after something goes wrong.

EU AI Act. High-risk systems must be tested before going on the market against metrics and thresholds set in advance, with declared accuracy and post-market monitoring over the system's life. The Digital Omnibus moved those duties for Annex III uses (credit, hiring, insurance pricing and others) to December 2, 2027. A support chatbot like the help-desk agent sits in the limited-risk tier: it must tell people they're talking to AI, and nothing in the Act tells it how to test. Separately, providers of the most capable general-purpose models have owed model evaluations and adversarial testing since August 2025, enforced from August 2, 2026. As of early October 2026 no AI Act fine had been found.

NIST and ISO. NIST's Generative AI Profile (AI 600-1, July 2024) is voluntary, and it says plainly that current pre-deployment tests "may be inadequate" and are often mismatched to how a system is used. Its suggested actions read like an eval checklist, from checking that each metric measures what it claims to evaluating in real-world conditions. ISO/IEC 42001 certifies a management system, so its auditor checks that you define verification criteria, keep evidence and monitor in operation, and never looks at your pass rate.

FDA and banks. The closest regulatory model of a good release gate is the FDA's predetermined change control plan for AI devices: the maker agrees in advance which changes it may make, how it will test them and what thresholds they must meet, and changes inside that envelope ship without a new submission. One study counted only 26 devices with authorised plans by May 2025. US bank regulators' April 2026 guidance excluded generative and agentic AI, so its ideas (independent challenge, outcome analysis, ongoing monitoring) describe a good eval program that no US bank is required to apply to an LLM.

Claims about quality. Eval numbers end up in marketing, and that's regulated. Texas settled with Pieces Technologies in 2024 over a claimed "critical hallucination rate" below 0.001%, and the SEC fined two investment advisers $400,000 combined in March 2024 for overstating their AI. A Stanford study found legal research tools marketed as "hallucination-free" made things up on roughly 17-33% of queries.

Customers. For a B2B seller, procurement bites before any law. AI addenda to security questionnaires ask whether the customer can pin a version, how much notice it gets of model changes, whether it can run its own evals beforehand, and how fast the vendor can roll back, while the vendor's own model provider may give it only 60 days' notice.

What mistakes cost

Let's say the help-desk company ships a prompt-and-model change that makes 5% of its million monthly conversations worse, and it takes three weeks before support tickets and satisfaction scores expose it. Rough arithmetic on stated assumptions:

• The damage: about 30,000-40,000 bad conversations. If each one means a person has to step in, at $2-6 a contact, that's $60,000-240,000, mostly paid first by the retailers' support teams and then by the help-desk company in credits and churn.
• What it gets back: nothing from its model provider. No SLA I found covers quality, accuracy is disclaimed, and any claim would be capped at a year of fees, about $54,000-162,000 here. General liability insurance increasingly excludes generative AI.
• What prevention costs: a regression suite of 500-2,000 cases run three times on a mid-tier judge costs about $20-300 per run, a human review of 100-200 judged cases $100-1,000, and keeping the suite current perhaps $30,000-90,000 a year in engineering time. One avoided three-week incident pays for the suite for roughly one to five years, depending on how often incidents happen.

The record so far:

The cash is small and falls on whoever ran the product; the large numbers are reputation and market value. Users, journalists or customers found almost all of these. In Anthropic's 2026 case, user reports started in early March, initially looked like normal variation, and the last fix landed on April 20, after the changes had passed internal evals and staff use. OpenAI's sycophancy update passed offline evals and an A/B test, and Google's AI Overviews passed red-teaming and live tests. Each of these teams had evals and lacked the right test case.