Becoming AI native: how your team ships, works and serves customers
For a product leader asked to make a regulated platform team AI native: measure first, then change how code ships, how internal work is reviewed and what customers see, in that order, with outcome metrics, real oversight and partner sign-off.
Last updated October 2026
The big picture
Say your CEO has read that Google says 75% of its new code is AI-generated, and asks you to make product and engineering "AI native". You run a platform where mistakes cost money, licences and partner trust: payments, cross-border payouts, telco or CPaaS, security or identity. Your engineers already use AI tools, some of them unapproved. Support and compliance are curious and nervous. Sales wants an AI agent in the product next quarter. Over about a year you need to change how the team ships code, how internal work gets done and what customers see, prove it worked, and keep quality, security and compliance intact.
I'd run it in four phases: baseline (measure first), ship code (with review), internal flows (AI assists people who decide) and customers (with sign-off). The baseline matters most, because developers can't feel the effect: in METR's July 2025 trial, experienced developers felt 20% faster with AI and were measured 19% slower. Without outcome metrics taken before rollout, you can't prove anything after it. One branch leaves the second phase and goes nowhere, and I call it theatre: usage without outcomes. Seats, prompts and leaderboards go up, and nothing customers or auditors care about changes.
Phases 2 to 4 each take one part of the job, in this order:
| Part | What changes | Phase |
|---|---|---|
| 1. How the team ships | Coding tools and agents, review, tests, security | 2 |
| 2. How internal flows work | Support, compliance, incidents, product work | 3 |
| 3. How customers are served | Assistants, agents, docs for agents, pricing | 4 |
The baseline (phase 1) measures all three before anything changes.
These ideas come from studies and other companies' accounts rather than my own rollout:
- AI speeds up writing code more than shipping it. Trials and field data agree on this one. The gains reach production only when work is well specified and tests, CI and review can absorb more change; otherwise they pile up as review queues, rework and instability. In open-source projects that adopted GitHub Copilot, core developers ended up reviewing more code and writing about a fifth less of their own. In projects that adopted Cursor, a burst of speed faded while warnings and complexity stayed up. Google's DORA research calls AI an amplifier of the system it lands in. See phase 2.
- AI native means changing how work and decisions get reviewed, much more than rolling out tools. I'm confident about this one too. Usage mandates and usage metrics get gamed: Amazon reportedly dropped an internal AI-usage leaderboard after staff pointed agents at pointless tasks to climb it (May 29, 2026). Oversight also has to be real, because under EU and UK rules a human who approves almost everything leaves the decision "solely automated". See Measuring it.
- Internal AI should start on high-volume, checkable work that splits into pieces, with AI assisting a reviewer. The best study of AI at work followed about 5,000 support agents: it helped novices a lot (34% more issues resolved per hour) and the most experienced hardly at all, so the gain is mostly in bringing new people up to speed. Emotional customer work and expert work need a different design. Klarna and the Commonwealth Bank of Australia both walked back AI-only support. See phase 3.
- On regulated platforms, partners gate customer-facing AI more than regulators do. Card networks, carriers, sponsor banks and enterprise security reviews decide what an AI may do with money, messages and data. Regulators draw a few hard lines: consent for AI voice calls, disclosure of AI under the EU AI Act from August 2, 2026, and human review of credit and KYC (identity check) decisions. The failures that actually cost companies money were quality failures, mostly chatbots that made up policies. See phase 4.
Other ideas sit where they apply: juniors and learning (phase 2); "AI researches, a human decides" and managers driving adoption (phase 3); buyers' data questions, docs that agents load, and outcome pricing (phase 4); where the law requires a human, by region (Where a human must decide).
The order matters: measure first, fix how you ship before generating more code, let AI gather and draft while named people decide, put AI in front of customers last, and judge it all by outcomes rather than usage.
What's different on a regulated platform:
- Separation of duties already applies to AI-written code. PCI DSS 6.2.3.1 says a manual code reviewer can't be the author. The EU's DORA change rules require independence between whoever builds a change and whoever approves it. An engineer who prompts an agent and then approves its pull request is, arguably, both.
- Partners can block customer-facing AI with no regulator involved. Under the 2023 US interagency guidance, a sponsor bank answers for a fintech's customer-facing tools. To a carrier, an AI agent that sends texts is ordinary business traffic.
- Some decisions need a human, and the rule differs by region. In the EU, onboarding and due-diligence decisions will need "meaningful human intervention" under the Anti-Money Laundering Regulation. In the US, credit decisions need specific, accurate reasons rather than a human.
- Someone else sets the dates: EU AI disclosure from August 2, 2026; software under EU product liability from December 9, 2026; EU high-risk duties from December 2, 2027.
The AI agent orchestration guide covers the agent technology, security and liability. This playbook is about leading the change.
01Weeks 1-4
Baseline
You have 3-6 months of outcome metrics for how the team ships and works, a one-page AI policy (approved tools, data rules, agent permissions), an agreed position with compliance on AI-written changes, and a staggered rollout plan.
Measure first, because people can't tell whether AI is helping them. In METR's trial, 16 experienced maintainers worked real issues in their own mature repositories, and with AI they took 19% longer, spending a good share of their time reviewing and cleaning up AI output. METR doesn't claim this generalizes, and says its February 2026 follow-up data are unreliable. What survives both is the gap between how it felt and what was measured, and if you skip the baseline, every later claim rests on that feeling.
Pull the delivery numbers with engineering, and don't make it a ranking. Start with the five metrics in DORA's guide (updated January 5, 2026), per team: change lead time, deployment frequency, recovery time after a failed deployment, change fail rate and rework rate. Split pull-request cycle time into coding, waiting for review, in review and waiting to deploy, so you'll see where AI moves the queue. Record PR size and each senior's review load. Add incidents, escaped defects, security findings, secrets detected, and code reverted or rewritten within 2-4 weeks. Compare each team with itself, never with other teams.
Do the same for the internal workflows you might change. For the support queue, KYB review (business identity checks), alert triage or incident write-ups, track weekly volume, handle time, backlog, reopen or error rate, QA pass rate, escalations and cost per case. Four weeks of data is enough. You'll need these numbers in phase 3, and you can't rebuild them afterwards.
Roll out in waves so the later teams become your comparison group. The best company evidence comes from that design: Microsoft, Accenture and a Fortune 100 company gave Copilot to random groups of developers, about 5,000 in all, and saw about a quarter more completed tasks, with juniors gaining most. I'd pick two or three pilot teams and bring the rest in after them.
Find out what AI is already in use. Most developers already use AI tools or plan to, per Stack Overflow's 2025 survey, and more than half of the workers BCG surveyed said they'd use unapproved tools if they weren't given approved ones. I think bans mostly push it underground. Samsung banned generative AI on company devices in May 2023, after engineers pasted source code and meeting notes into ChatGPT. Approved tools good enough to beat the shadow ones are a security control.
Write a one-page AI policy. Only a quarter of US organizations had communicated a clear AI strategy in Gallup's May 2026 data, and DORA counts a clear, communicated AI stance among its seven AI capabilities. Mine would cover:
- Approved tools on enterprise terms: no training on your code, stated retention and region, SSO, admin audit logs, IP indemnity.
- Data classes: what may go into a prompt or an agent's context. Never card data, production credentials, KYC files or partner-confidential material.
- Secrets and dependencies: secret scanning before commit and in CI; lockfiles and allow-listed registries; check any package an AI suggests.
- Agent permissions: no production write access, dev and prod kept apart, a human for destructive commands, and each agent with its own identity.
- Attribution: tag AI-assisted changes. The Linux kernel's policy (December 23, 2025) bars agents from certifying a contribution and asks for an
Assisted-bytag; the human submitter is fully responsible. - Prototype line: PM and designer prototypes live in sandboxes, and anything promoted to production goes through normal review.
- IP: the US Copyright Office (January 29, 2025) says prompts alone don't make you an author, but human selection, arrangement or editing can.
Agree the compliance position before the first agent PR. Take PCI DSS 6.2.3.1, SOC 2 CC8.1 and the EU DORA rule (Art. 17) to compliance and your assessor. Ask who the author is when an engineer prompts an agent, what record they expect per change, and whether an AI reviewer counts as review. I found no auditor or regulator guidance written for AI-written code. My view is that the person who prompted it is the author and can't be the only approver, but test that with them.
Pick the scorecard now, before anyone has a result to defend. It's in Measuring it: adoption, flow, guardrails, value and people, reviewed monthly by you, the engineering lead and someone from compliance.
Checklist
- Pull 3-6 months of DORA's five metrics per team with engineering, and split PR cycle time into its four segments
- Record PR size, senior review load, incidents, escaped defects, security findings, secrets and 2-4 week churn
- Baseline volume, handle time, reopen rate, escalations and cost per case for each candidate internal workflow
- Run a short developer-experience survey: cognitive load, flow, current AI use and trust in its output
- Inventory the AI tools already in use, approved or not
- Publish the one-page AI policy: tools, data classes, secrets, dependencies, agent permissions, attribution, prototype line
- Agree with compliance and your assessor how AI-written changes meet author-not-approver and audit-trail rules
- Confirm CI gates: tests, static analysis, dependency and secret scanning, a PR size limit, required human approval
- Choose two or three pilot teams and a staggered rollout so later teams act as the comparison
- Define the scorecard and who reviews it monthly
02Months 1-3
Ship with AI
Coding assistants and agents are in daily use on the pilot teams, starting with bounded work; review, tests, security and permissions keep pace; and at month 3 you decide with evidence whether to expand, fix the delivery system or stop specific uses.
Expect the bottleneck to move to review. In one vendor's study of more than 10,000 developers, teams that used AI heavily merged almost twice as many PRs, but review time nearly doubled too, and the company as a whole got no faster (July 2025). Google saw the same in large AI-assisted migrations: total time fell by about half, and then review and rollout became the limit instead of writing code (January 2025). DORA's 2025 survey linked AI to more throughput and more instability, and its 2026 ROI report expects a dip before returns. Throughput only counts if change fail rate, rework and incidents hold. Take a payments team that goes from 40 merged PRs a week to 70, while the typical PR now waits three days for review instead of one and more deploys get rolled back. That's more code, and I wouldn't call it faster delivery.
I'd start agents on bounded work where CI already says what "done" means, because that's where the clearest results came from:
- Stripe's in-house agents ("Minions") open more than 1,300 PRs a week. A human reviews every one, and they're submitted only after CI passes. They're best at config changes, dependency upgrades and small refactors (InfoQ, early 2026). For a regulated platform, it's the closest analogue I found.
- Spotify's background agent had more than 1,500 merged PRs by November 2025 and saved most of the time on migrations. It sits on a fleet-management system that already automated about half of PRs before AI.
- Amazon claimed in August 2024 that AI-assisted Java upgrades saved about 4,500 developer-years.
My first list for a platform team would be dependency and CVE (known vulnerability) upgrades, migrations, flaky-test fixes, lint and config sweeps, and SDK and docs regeneration. They're high volume, have an objective "done", and do little damage if they go wrong.
Before you generate more code, strengthen the checks. Spotify's agent works because of its feedback loops: formatters and linters as tools, tests, a model judging the diff, then human review. Add contract tests for partner APIs and test coverage on payment, messaging and identity paths. Enforce a PR size limit; DORA has long tied small batches to stability, and it's one of its seven AI capabilities.
An AI reviewer is useful as a filter, but it can't be the approver. Microsoft runs an AI reviewer on more than 90% of its pull requests (July 14, 2025). That measures coverage rather than outcomes, and a regulated change still needs an accountable human. On PCI- or DORA-scoped code, the person who prompted the change never approves it. Record per change: the ticket, which tool or agent and model, who approved, test and scan results, the deploy record, the rollback path, and the agent's identity and permissions at the time.
On security, the code tends to be weaker and the tools are themselves a way in. In a Stanford trial, people with an AI assistant wrote less secure code and were more sure it was secure (2023, older models). A vendor benchmark found raw model output picked an insecure option in almost half of tasks. Models also invent package names, and often the same ones each time, so attackers can register them first. One vendor found commits co-written by a coding agent leaked secrets at about twice the usual rate. The tools themselves have shipped prompt-injection flaws, and the AI agent guide explains why there's no complete fix.
Don't give agents production write access. In July 2025 Replit's agent deleted a production database during a declared code freeze, and Replit then separated dev and prod. The Financial Times reported a 13-hour outage of an AWS service in one China region in December 2025, after Amazon's Kiro agent chose to delete and recreate an environment; Amazon blamed misconfigured access controls rather than AI. Either way the fix is least privilege and separation of duties, which your audits already require.
Protect your senior reviewers, since they pay the review tax, as the Copilot study above showed. Rotate reviewers, set review service levels and watch review load weekly.
Keep juniors learning. They gain most in output, but in an Anthropic trial, mostly junior engineers learning a new library with AI scored 17% lower on a mastery quiz, worst at debugging, and weren't much faster. Payroll data show employment of 22- to 25-year-olds in AI-exposed jobs, software included, fell about 13% relative to others from late 2022, though a counter-study finds no economy-wide disruption yet. My rule: no "accept without understanding" on payments, messaging or identity code. Pair AI use with debugging and code-reading exercises. And keep hiring juniors, because they're your future reviewers.
Budget per merged change rather than per seat. Anthropic's Claude Code docs put enterprise spend at a few hundred dollars per developer a month, a fair planning anchor for one agentic tool.
Shopify (April 2025) made AI use a baseline expectation, part of performance reviews, and something teams had to rule out before asking for headcount. Coinbase (August 2025) fired engineers who hadn't onboarded to AI tools within a week. Amazon set a weekly usage target in November 2025, then dropped its usage leaderboard after it was gamed and moved to "normalized deployments". None of them has published an effect on outcomes. I'd treat usage as a team-level input, pair it with outcome metrics from day one and keep it out of individual ratings; otherwise you're on the theatre branch.
Decide at month 3 with the scorecard, and expect a dip in the first month or two.
Checklist
- Roll out approved tools to the pilot teams on enterprise terms, with SSO and admin logs, and track spend per merged change
- Start agents on dependency and CVE upgrades, migrations, flaky tests, lint and config sweeps, SDK regeneration
- Run agents in sandboxes with no production credentials, scoped short-lived tokens and their own identity
- Tag every AI-assisted PR; on regulated paths, the person who prompted a change never approves it
- Enforce PR size limits and add an AI pre-reviewer as a filter, not an approver
- Protect senior review capacity: rotation, review service levels, weekly review-load numbers
- Add contract tests for partner APIs and coverage on critical paths before scaling generation
- Pair juniors' AI use with debugging and code-reading work; keep hiring juniors
- Keep usage metrics at team level and out of individual reviews
- Decide at month 3: expand, change the system, or stop specific uses
03Months 2-6
Rewire internal flows
One or two high-volume internal workflows are rebuilt so AI does the gathering and drafting, named people make the decisions the rules and the business require, oversight is measured to be real, and the change shows in an outcome metric.
Pick the workflow with a scorecard rather than on enthusiasm. Use the seven criteria in Choosing the first workflows: volume, checkable output, splits into pieces, lots of novices, written context, low emotional temperature, reversible. Every positive result I've seen is on high-volume work.
Design for novices, and leave experts room. In the support study, the assistant spread top performers' know-how to new agents and barely helped the most experienced. A trial at Alibaba's Taobao after-sales desk found the same shape: agents spotted issues about a third faster, but objective quality didn't change, and the best agents got slower (2026). On a partner-heavy platform with long onboarding (numbering rules, card disputes, cross-border corridors), that ramp time is where the money is. Don't force the assistant on your best people.
The pattern I'd copy is "AI researches, a human decides". Stripe's compliance team splits each review into small sub-tasks. Research agents fetch the evidence for each one, the human reviewer must answer each one, and every agent action and its reasoning is logged for examiners. Median handling time fell about a quarter (June 26, 2026), though that's the company's own account, co-published with its cloud vendor, with no control group or failure data. The reviewer ends up working more like an editor, and accountability stays where it was.
Picture a KYB team that reviews 300 new merchants a week. For each one, the AI pulls the company registry entry, the owners and any sanctions hits, and the analyst answers each question and signs off. Other published results fit the pattern. Google's security team had AI draft incident summaries for people to accept, edit or discard, and cut writing time by about half. Meta's root-cause tool gives engineers a shortlist of five, and the real cause is on it less than half the time. HSBC's big gains in transaction monitoring came from machine-learning risk scoring rather than a language model (2024). For language models in compliance, the evidence is still thin.
Route by risk and emotion before the AI touches the case. In a second Alibaba experiment, AI handled eligible chats while humans monitored, across hundreds of thousands of chats in August 2024. Chats got faster, ratings on AI-handled chats fell a lot, and when frustrated customers were escalated after a bad AI exchange, people mostly couldn't win them back (July 9, 2026). So I'd send angry, high-value or legally sensitive cases to people first.
Make oversight real, and measure it. People defer to machines. When radiologists got wrong AI suggestions, even those with 15+ years' experience dropped from about 80% accuracy to under half (2023). Developers approved almost all of an AI coding tool's permission prompts in one lab's telemetry (from the site's AI agent research). And people lose the skill: endoscopists found fewer growths in procedures without AI after three months of using it (an observational study). So I'd set it up like this:
- Reviewers see the sources, can override cheaply, and are measured on accuracy rather than throughput.
- Track approval rate and time per review. An approval rate near 100%, or reviews in seconds, should trigger a sampling audit, because under EU and UK law that's evidence the decision is still solely automated (see Where a human must decide).
- Seed known-wrong cases into the queue each month to measure how often people just agree.
- Keep a no-AI path: juniors do some cases unaided each week, and seniors review them.
If a reviewer approves 99% of AI-drafted onboarding decisions in about eight seconds each, there's a human on paper but the machine is deciding.
Write the context down first. Anthropic found complex business deployments are held back by getting scattered company knowledge into context rather than by the model (September 2025).
In product work, Procter & Gamble found individuals with AI matched two-person teams without it. BCG consultants with AI were faster and did more on tasks inside AI's "jagged frontier", but were 19 points less likely to be right on a task outside it. Spec drafting and synthesis are usually inside; new judgment calls on thin data can be outside. Keep synthetic users out of discovery readouts. In one study, AI "respondents" flipped the direction of a real survey finding about a third of the time. "Ask the data" works on curated data layers. On real company schemas, the best agent solved about a fifth of tasks in November 2024, though models have improved since.
Watch for gains that disappear into checking. Among 25,000 Danish workers in exposed jobs, chatbots saved about 3% of time, with no effect on earnings or hours, because new checking work absorbed the gains (2025).
Managers and training seem to drive adoption. Gallup found people whose manager supports AI use were almost twice as likely to use it weekly. BCG found five or more hours of training, in person with coaching, went with regular use, and only about a third of workers felt trained enough. Both are only correlations. Still, I'd train your managers first and count training in hours.
Be clear about capacity before launch. The Commonwealth Bank of Australia reversed 45 job cuts tied to a voice bot after its union challenged the numbers (August 2025). Duolingo's CEO admitted an "AI-first" memo hadn't given enough context. Tell the team whether freed time goes to the backlog, to other work or to slower hiring, and involve worker representatives where they exist. In the EU, AI that assigns work to employees or evaluates them is high-risk from December 2, 2027.
Checklist
- Score candidate workflows on the seven criteria and pick one or two scoring at least 10 of 14
- Map every decision in the workflow to its rule: AI may draft, AI may recommend, a human must decide, a human must explain
- Decompose the case into sub-tasks; AI researches each; the reviewer answers each; log everything
- Build an eval set of 50-200 real past cases with the people who do the work, and pass it before go-live
- Route by risk and sentiment before the AI touches a case
- Measure reviewers on accuracy; track approval rate and time per review; seed known-wrong cases monthly
- Keep some cases done unaided by juniors and reviewed by seniors
- Train managers first and count training in hours
- Tell the team what happens to freed capacity before launch
- Set the outcome metric and the rollback trigger up front; review at 6 and 12 weeks
04Months 4-12
Put AI in front of customers
Customers can use your platform through their own agents, your first customer-facing assistant is live with disclosure, evals, a fast human path and partner and regulator sign-off, and buyers' security reviews and pricing questions have written answers.
Go in order of risk, lowest first. My default is to make the platform easy for developers' agents first (months 4-5; low exposure, and it forces API quality work), then a read-only assistant over docs and account data (months 5-8), then a copilot that drafts changes for a human to apply (months 8-12, after partner sign-off). Autonomous actions on money, end-user messages or decisions about people come last, only inside partner rules, and with human review where the law requires it.
Get partners' sign-off criteria before you build, since they set the gates where money and messages move:
- Card networks: Mastercard Agent Pay, Visa Intelligent Commerce (both April 2025) and Visa's Trusted Agent Protocol (October 2025) route agent payments through tokens and registration. Mastercard has reportedly published rules for agent payments; I haven't read them, and who eats the chargeback when an agent buys the wrong thing is still open.
- Carriers: US business texting needs 10DLC brand and campaign registration, and T-Mobile's code of conduct fines senders up to $2,000 per violation and blocks traffic. I found no carrier rule specific to AI-written messages; AI texts are just business traffic. See the telco guide.
- Sponsor banks: they answer for customer-facing tools under the 2023 guidance, and their model-risk teams may still review generative AI even though the April 2026 US model-risk guidance leaves it out.
- Enterprise buyers: most B2B decision-makers in a vendor's survey said their security, legal and compliance teams now evaluate AI more strictly (April 2025).
Regulators draw only a few hard lines. On disclosure, under EU AI Act Article 50, from August 2, 2026, people must be told they're talking to an AI. The Commission's July 2026 guidelines say at the very first interaction, and a generic "assistant" label or text buried in terms won't do. A CPaaS or support platform is often the provider and its customer the deployer, so I'd make disclosure the default and hard for customers to remove. The US has no federal chatbot-disclosure law, only state rules such as Utah's and Maine's. On voice, in February 2024 the FCC ruled that AI voices count as "artificial" under the TCPA, so outbound AI calls need prior express consent, and Lingo Telecom paid $1M for carrying deepfake robocalls. On decisions, no AI-only rejections on credit, KYC or access (see Where a human must decide).
Spend most of the effort on quality, because that's what has hurt companies. Air Canada's and Cursor's bots made up policies (see Examples), and Klarna's CEO said a focus on cost had produced lower quality (May 2025). Customers tend to leave well before anything reaches a court. These are the pieces I'd build:
- Grounding: answers about policy, prices, terms and limits come only from approved sources. Otherwise the bot says "I don't know" and hands off. If a merchant asks whether the fee on a failed payout gets refunded and that policy isn't in the approved sources, the bot hands over to a person instead of guessing.
- A fast human path: the US consumer-finance regulator's 2023 chatbot report warned against "doom loops" and expected a route to a human. It's unclear whether it still stands behind that after its 2025 guidance withdrawals, but your customers will. Measure time to a human.
- Evals from real tickets: start with 20-50 tasks, graded on the account's end state (refund issued once, rule deployed correctly), and report how reliably they pass across repeated runs. On one benchmark, customer-service agents passed all eight tries on fewer than a quarter of tasks (2024, from the site's AI agent research). Pin model versions and re-run evals on every model, prompt or tool change.
- Write actions: scoped short-lived credentials, approval for irreversible steps, spend caps, idempotency (a repeated request doesn't act twice), a kill switch and a full audit log.
Every number in your marketing needs a dated eval behind it. The FTC fined accessiBe $1M over a claim that its AI made any site accessible, went after Workado for claiming 98% accuracy that testing didn't support, and sued Air AI over AI sold as a replacement for sales reps.
Have the buyer's security answers written before sales needs them. Buyers have moved from "can we opt out of AI?" in 2024 to asking you to prove their data never trains your models (2026, a vendor report). Zoom (August 2023) and Slack (May 2024) both faced backlash over terms read as allowing training on customer content. For B2B buyers the default and the opt-out are part of the product, so ship training off by default, an admin toggle, a public page on how you use AI and data, and a sub-processor list naming your model providers. Prepare answers to the Cloud Security Alliance's AI questionnaire (AI-CAIQ) and a view on the ISO/IEC 42001 AI management standard. Don't promise customers more notice of model changes than the labs give you, which ranges from 60 days to six months.
Build docs for the agents that actually read them. More and more of your customers reach your API through their own agents. Vercel tested coding agents on Next.js APIs they hadn't seen in training. With a compressed docs index in the project's AGENTS.md file they passed every task, far more than with no help or with a separate skill the agent often never called (January 2026, a vendor test that doesn't give the task count). Since version 16.2, Next.js ships its docs as Markdown inside the package. By contrast, server logs across 137,000 domains showed 97% of llms.txt files got no requests at all in May 2026, so llms.txt is cheap to ship but I wouldn't count it as a strategy. My agent-ready list, following Stripe's Markdown docs and official MCP server and Anthropic's tool guidance: a complete OpenAPI spec with examples and error meanings; Markdown docs, bundled with SDKs where you can; a remote MCP server (the standard way agents connect to tools) with OAuth, scopes and read-only by default; a sandbox, idempotency keys and dry runs; short-lived keys for agents; and evals of common integration tasks on every docs and API release. Watch the cost, too. In Twilio's small test of its early MCP server, tasks were faster but cost about a quarter more (three tasks, April 2025).
Price on a unit you can audit. Intercom charges about a dollar per outcome, and an outcome includes a customer asking for no further help. Zendesk introduced per-resolution pricing in August 2024, and HubSpot reportedly moved to per-resolution pricing in April 2026. "Resolved" is the vendor's definition, so the disputes move to the definition. My default is to fold AI into your existing per-transaction or per-message price where it lowers your cost to serve, charge for usage where each call costs you compute, and use outcome pricing only where the outcome is objective and auditable. Take an SMS platform that adds AI to draft replies to inbound texts. If it mostly cuts the platform's own support cost, keep the per-message price; if each draft costs real model compute, charge per draft. Put tokens, retries and the human fallback in the margin model.
Check liability and insurance before launch. If your AI screens, scores or decides for your customer, you may share the liability (Mobley v. Workday), and the revised EU Product Liability Directive treats software, AI included, as a product from December 9, 2026 (both from the site's AI agent research). Check your errors and omissions insurance and cyber cover for AI exclusions.
Checklist
- Pick the first use case by risk tier and write its intended use and "never does" list
- Ship agent-ready docs, a clean OpenAPI spec and a read-only MCP server with OAuth before any customer-facing agent
- Show sponsor bank, acquirer, carrier and key customers the design before build, and get written sign-off criteria
- Turn disclosure on by default at the first interaction, in text and voice; customers can't silently remove it
- Cover AI in consent language for outbound voice and text; register the campaign; honour opt-outs
- Publish the AI data-use page and sub-processor list; training off by default with an admin toggle
- Prepare the security pack: AI questionnaire answers, ISO/IEC 42001 plan, prompt-injection threat model, red-team results
- Run evals from 20-50 real tickets, graded on end state across repeated runs; pin model snapshots
- Keep a visible, fast human path and measure time to a human
- Review every marketing number against a dated eval; price on an auditable unit
Measuring it
Outcome metrics prove the change, and usage metrics only show activity. Goodhart's law kicks in right away: when a measure becomes a target, it stops being a good measure. The scorecard I'd use has five layers and is built on DORA. Review it monthly, at team level, and keep it out of individual ratings:
| Layer | Measures | Watch for |
|---|---|---|
| Adoption | Weekly users, AI-tagged PRs, agent PRs merged, spend | An input; never a target |
| Flow | DORA throughput; PR cycle segments | Review wait growing |
| Guardrails | Change fail, rework, incidents, security findings, churn | Must hold for flow to count |
| Value | Roadmap items delivered, backlog burned, cost per change | The question the CEO asked |
| People | Developer experience, trust, junior skill checks | Seniors' review load |
Internal flows and customer-facing AI work the same way, with quality measured right next to speed. Handle time improved in every support study above, and objective quality often didn't. Measure reopens, retrials, escalations, wrong answers on audited samples, complaints, refunds issued by AI and cost per resolved case.
| Vanity metric | Why it misleads | Use instead |
|---|---|---|
| % of code written by AI | Volume; definitions vary | Lead time with change fail rate |
| Suggestion acceptance rate | An autocomplete UX number | Rework within 2-4 weeks |
| Seats, prompts, tokens | Activity; gamed by leaderboards | Cost per merged change |
| Handle time | Speed without quality | Reopens and retrials |
| "Resolution" rate | The vendor's billing event | Audited correct end state |
| FTE equivalent | Not a cost line | The cost line itself |
I'd treat the labs' code-share figures as claims rather than benchmarks. Google's CEO said 75% of new code was AI-generated in April 2026, counting AI-suggested code that engineers accept. Microsoft's CEO said 20-30% in April 2025. Anthropic said Claude wrote more than 80% of its merged code in May 2026. OpenAI said almost all its employees used its Codex agent by June 2026, which is a usage figure rather than a code share. None of them publishes change fail rates or incidents next to these numbers, and one analyst estimated that a code share near 90% could come with well under twice the productivity.
To keep it honest, pair every throughput metric with a stability metric, publish the definitions, check a sample of PRs and cases by hand, and watch early signs (review queue age, PR size, approval rate) weekly and lagging ones (incidents, complaints) monthly.
Where a human must decide
Where the law actually requires a person differs by region, and in the US it's narrower than most teams assume. This is how I read it as of October 2, 2026. I'm not a lawyer, so take it to counsel.
Credit, incl. small business
- US
- Specific, accurate reasons (Reg B)
- EU
- Person: high-risk from 2027-12-02
- UK
- Safeguards (from 2026-02-05)
Onboarding, due diligence
- US
- Sound AML programme, no rule per case
- EU
- Meaningful human intervention (AMLR)
- UK
- Safeguards
Suspicious activity reports
- US
- Institution decides and files
- EU
- Institution decides
- UK
- Institution decides
Fraud detection
- US
- No specific rule
- EU
- Excluded from AI Act high-risk
- UK
- Safeguards if significant
Hiring, evaluating staff
- US
- Notice in Illinois from 2026-01-01
- EU
- High-risk from 2027-12-02
- UK
- Safeguards
Any significant automated decision
- US
- State laws (Colorado from 2027-01-01)
- EU
- GDPR Art. 22: human or exception
- UK
- Safeguards, not a ban
Customer talks to an AI
- US
- State disclosure laws
- EU
- Disclose from 2026-08-02
- UK
- No AI-specific rule
Outbound AI voice call
- US
- Prior express consent (TCPA)
- EU
- Disclose (Art. 50)
- UK
- No AI-specific rule
United States. The US mostly asks for reasons rather than a reviewer. Under Reg B, a rejection notice must give the specific main reasons. "Internal standards" or a failed score isn't enough, and it applies to business credit too. If a model helps decline a merchant or a small-business credit line, it has to be able to produce those reasons. When a model declines a small café's credit line mostly because of thin, uneven cash flow, the notice has to say that, and "did not meet our criteria" won't do. The CFPB withdrew its circulars on complex algorithms in May 2025 "pending review", but the law and Reg B didn't change. For anti-money-laundering, the Bank Secrecy Act asks for a programme "reasonably designed" and doesn't require a human on every alert. FinCEN's October 2025 FAQs eased the paperwork for suspicious activity reports, and a 2018 joint statement says a pilot that finds more suspicious activity won't by itself prove your existing process was deficient.
European Union. The EU asks for human intervention. GDPR Article 22 gives people a right not to be subject to solely automated decisions with legal or similar effect, and the SCHUFA judgment (December 2023) says a score that plays a deciding role counts. The Anti-Money Laundering Regulation's Article 76(5) requires meaningful human intervention in automated decisions to accept, refuse or change due diligence on a customer. It covers companies too, so an EU platform's KYB decisions are in scope. It's reported to apply from July 10, 2027, but I'd confirm that date. The AI Act makes credit scoring of people, and AI that hires or evaluates staff, high-risk from December 2, 2027, with oversight that guards against people just agreeing with the machine.
United Kingdom. From February 5, 2026, the UK allows significant automated decisions with safeguards, and "no meaningful human involvement" is what makes a decision solely automated. The FCA has no AI-specific rules; Consumer Duty and the senior-manager regime apply.
In all three, the word that matters is "meaningful". A person who rubber-stamps a score doesn't make the decision any less automated, and approval rates near 100% and seconds per case are evidence against you. Australia's Robodebt scheme raised about A$1.7B of unlawful debts against more than 400,000 people while humans followed the flag.
Choosing the first workflows
Score each candidate 0-2 on seven criteria and pick one or two that score at least 10 of 14. It's my own scorecard, built from the studies above, so treat the threshold as a starting point.
| Criterion | Scores 2 when | Evidence |
|---|---|---|
| Volume and repetition | Hundreds of cases a week with a familiar shape | Every positive result |
| Checkable output | A right answer exists and checks fast | Support studies |
| Decomposable | The case splits into sub-tasks to research | Stripe compliance |
| Novice-heavy, long ramp | New staff take months to get good | +34% for novices |
| Context written down | Policies and past cases are documented, permissioned | Anthropic; Spider 2.0 |
| Low emotional temperature | Analyst-facing, not angry-customer-facing | Alibaba agentic trial |
| Reversible, contained | Errors are caught before a customer or regulator | Score 0 if AI acts ungated |
Say you're looking at alert triage on a payouts platform: thousands of alerts a week, a clear right answer for most, analysts who take months to get good, and nothing reaches a customer before an analyst decides. That scores high, and a queue of angry merchants disputing frozen funds wouldn't.
Good first candidates: KYB and due-diligence research, draft alert triage, incident summaries, draft replies for internal support agents, and partner-integration troubleshooting.
I wouldn't start with any of these:
- EU onboarding or due-diligence refusals without a design for meaningful human intervention
- Any credit decision on people or small businesses without specific reason codes
- Evaluating or monitoring employees in the EU
- Customer-emotional queues where the AI answers first
- A workflow whose owner won't commit reviewer time
- A headcount target set before the eval results
Examples
Stripe: agents on bounded code, research for compliance
Stripe's coding agents open more than 1,300 PRs a week, each passing CI and a human review. In compliance, agents gather the evidence and reviewers decide, and handling time fell about a quarter. Both are Stripe's own accounts. The wins came from work with a clear "done" and a human still deciding (phase 2, phase 3).
Spotify and Google: migrations where CI defines "done"
Spotify's background agent had merged more than 1,500 PRs by late 2025, mostly migrations. Google's AI-assisted migrations took about half the time, review included. Both built on test loops they already had, and review and rollout set the pace more than the AI did (phase 2).
Amazon: a usage leaderboard, gamed
Amazon set a weekly AI-usage target with an internal leaderboard. Staff pointed agents at pointless tasks to climb it, and Amazon dropped it in May 2026. I'd keep usage as an input on your scorecard and never make it the goal (Measuring it).
Klarna: from AI-first support back to an easy human
In 2024 Klarna said its assistant did the work of 700 agents. By May 2025 its CEO said human support quality was the way forward, and it started hiring people again. Klarna kept the AI and dropped AI-only, so I'd trust the cost line over the output count (Measuring it).
Commonwealth Bank of Australia: the claim the union disproved
The bank cut 45 support jobs, saying a voice bot had reduced calls. The union showed call volumes were rising, and the bank reversed the cuts and apologized. Without a baseline it couldn't defend the capacity claim, so measure before any headcount decision (phase 1).
Alibaba (Taobao): assist versus act
Alibaba tested AI two ways in after-sales chat. Helping people, it lifted the weaker agents most. Acting on its own with people watching, it made chats faster but customers less happy. AI helping a person and AI acting while a person watches are different designs (phase 3).
Air Canada and Cursor: invented policies
Air Canada's chatbot made up a refund policy, and a tribunal held the airline liable (more here). Cursor's support bot made up a one-device rule, customers cancelled, and Cursor started labelling AI replies. In both cases the company owned what its bot said, so ground policy answers in approved sources and label the AI (phase 4).
Zoom and Slack: the training default
Zoom's 2023 terms were read as allowing AI training on customer content, and Zoom reversed them within days. Slack faced the same backlash in 2024 over an email-only opt-out. For business buyers, the default and the opt-out are part of the product (phase 4).
What usually goes wrong
| Failure | Early sign | Case or evidence |
|---|---|---|
| Rollout without a baseline | Success measured by survey | METR |
| Usage theatre | Leaderboards; AI usage in individual reviews | Amazon |
| Review collapse | PR size and review wait climbing; seniors' output down | Faros; Xu et al. |
| Agent with too much power | Production credentials in an agent's reach | Replit; reported AWS outage |
| Prompter self-approves | No record of which agent did what | PCI 6.2.3.1; DORA RTS |
| Rubber-stamp oversight | Approval near 100%, reviews in seconds | Radiology; SCHUFA |
| Handle time up, quality flat | Only speed on the dashboard | Alibaba |
| AI-only support, then reversal | Headcount target before eval results | Klarna; CBA |
| Hollowed-out junior pipeline | No junior hires; juniors only edit drafts | Canaries; Anthropic trial |
| Invented policy to customers | No grounding, no label | Air Canada; Cursor |
| Training-default surprise | Opt-out by email | Zoom; Slack |
| Partner surprise | AI voice or texts launched without carrier review | Lingo; T-Mobile fines |
| Docs theatre | llms.txt shipped; API spec and errors unchanged | Ahrefs logs |
The most common failure in these accounts is buying tools while everything else stays the same. DORA's "amplifier" finding cuts both ways: good tests, small batches and clear ownership get better with AI, and weak ones get worse faster. Every failure in the table is a review or measurement failure before it becomes a technology one.
Words that mean something else here
| Term | What you'd assume | What it means here |
|---|---|---|
| "% of code written by AI" | Productivity | Volume, defined differently by each company |
| AI native | Everyone uses AI tools | Work and decisions reviewed differently, proven by outcomes |
| Human in the loop | Compliant oversight | Only if the human is competent, informed and can say no |
| Faster | Measured speed | Often felt speed; METR found a 39-point gap |
| Resolution | The customer's problem fixed | The vendor's billing event, which can include silence |
| llms.txt | Agents read your docs | A file format; 97% got no requests |
| MCP support | Production-ready | Anything from an alpha local server to remote with OAuth |
| No training on your data | Covers everything | May cover the model provider, not your own models |
| AI code review | The review auditors expect | A filter; an accountable human still approves |
| Agent | One thing | Autocomplete, chat, in-editor agent or background agent |
| DORA | One thing | Delivery metrics, or the EU resilience act; both apply |
| The law requires a human (US) | A reviewer per decision | Mostly specific reasons and a sound programme |
| FTE equivalent | Cost saved | Klarna's count rose while its cost line rose too |
On DORA. On a payments or banking platform, the two meet in change management. Google's DevOps Research and Assessment metrics measure it, and the EU's Digital Operational Resilience Act, in force since January 17, 2025, regulates it. In a meeting with engineering and compliance, say which one you mean.
Sources
Read on or before October 2, 2026 unless dated; "search result" means I saw it only as a snippet or summary. Vendor studies are marked in the text.
Controlled and quasi-experimental studies
- Coding: METR 2025 and paper; METR update 2026 (unreliable per METR); Cui, Demirer et al. and in Management Science; Anthropic skill-formation trial; Perry et al.; He et al.; Xu et al.; AIDev dataset; package hallucinations, via CSA
- Work and support: Brynjolfsson, Li & Raymond; Ni et al.; Alibaba agentic study and Tuck summary; Humlum & Vestergaard and via NBER; Dell'Acqua et al., P&G; BCG jagged frontier, via the Crimson; Bisbee et al.; Spider 2.0; Microsoft Security Copilot trials (vendor-run)
- Oversight and skills: Dratsch et al., via RSNA; Budzyń et al.; Canaries in the Coal Mine and follow-up
Surveys and analyst figures
- DORA 2025, DORA metrics guide, DORA capabilities, DORA ROI via InfoQ; Stack Overflow 2025 via InfoWorld; Gallup; BCG AI at Work 2025
- Vendor studies: Faros 2025, Faros 2026, GitClear via DevClass, Veracode, GitGuardian, DX framework, G2 2025, SecurityPal 2026, Vercel agent evals, Twilio MCP test; llms.txt logs, via PPC Land
Company reports and claims
- Labs: Google 75%, via Semafor; Google migrations; Google incident summaries; Microsoft 20-30%, via TechRadar; Microsoft AI code review; Anthropic 80%, via Fortune; Anthropic Economic Index; Anthropic tool guidance; Claude Code costs; OpenAI Codex paper; Meta root-cause analysis; Greenblatt, Redwood
- Coding at other companies: Stripe Minions and via InfoQ; Spotify part 1, part 3 and via InfoQ; Amazon Java upgrades, via The Decoder; Amazon leaderboard, via The Decoder; Replit, via the AI Incident Database; AWS outage, via GeekWire; Shopify, via Digital Commerce 360; Coinbase, via TechCrunch; Samsung, via Fortune
- Internal work and support: Stripe compliance, via AWS; HSBC; Klarna via Mind the Product, Maginative and CX Dive; Commonwealth Bank of Australia via ACS Information Age and Bloomberg; Duolingo, via Fortune; Cursor, via The Register
- Customers, docs and pricing: Visa Intelligent Commerce; card networks, via PYMNTS; Stripe docs for LLMs and Stripe MCP; Next.js 16.2; Intercom pricing; Zendesk; HubSpot (search result); Zoom, via ISACA; Slack, via TechCrunch
Regulators, law and standards
- Change control: PCI DSS 6.2.3.1 (mirror); EU DORA RTS Art. 17; SOC 2 CC8.1 and agents (practitioner); Linux kernel policy; US Copyright Office
- Decisions: Reg B §1002.9; CFPB withdrawn guidance; FinCEN 2018 joint statement; SAR FAQs 2025; SCHUFA; AMLR Art. 76; AI Act Art. 14 and Annex III; UK Data (Use and Access) Act; Robodebt Royal Commission
- Customers: EU Art. 50 guidelines; FCC AI voice ruling; FCC NPRM; Lingo Telecom; Utah; Maine; 2026 state chatbot laws; third-party guidance; SR 26-2; CFPB chatbot report, summary; FTC on accessiBe, Workado, Air AI; CSA AI Controls Matrix; carrier codes of conduct
The site's own research
- AI agent orchestration (agent security, evals, liability, regulation; approval telemetry, tau-bench, Air Canada, Mobley v. Workday, model retirement notice), telco, order-to-cash; Your first 60 days leading product for DORA metrics and reading the team
Examples