Becoming AI native: how your team ships, works and serves customers

Measure first, because self-report can be wrong in sign. METR's 16 experienced maintainers worked 246 real issues in their own mature repositories; with AI they took 19% longer (confidence interval +2% to +39%), accepted fewer than 44% of generations and spent about 9% of their time reviewing and cleaning AI output. METR explicitly doesn't generalize the result, and its February 2026 follow-up was compromised by selection effects, which METR itself says make the data unreliable. The lesson that survives both is the gap between feeling and measurement. If you skip the baseline, every later claim rests on how people feel.

Pull the delivery numbers with engineering, not as a ranking. DORA's five metrics per team: change lead time, deployment frequency, failed-deployment recovery time, change fail rate and deployment rework rate (DORA guide, updated January 5, 2026). Split pull-request cycle time into coding, waiting for review, in review and waiting to deploy, so you'll see where AI moves the queue; record PR size and each senior's review load. Add incidents, escaped defects, security findings, secrets detected and code reverted or rewritten within 2-4 weeks. Compare each team with itself, never with others.

Baseline the internal workflows you might change. For the support queue, the KYB review queue, alert triage or incident write-ups: weekly volume, handle time, backlog, reopen or error rate, QA pass rate, escalations and cost per case. Four weeks is enough. You'll need these in phase 3, and they're impossible to rebuild afterwards.

Plan a staggered rollout, so later teams are your comparison group. The strongest company evidence comes from that design: field experiments with 4,867 developers at Microsoft, Accenture and a Fortune 100 company randomized access to Copilot and found 26% more completed tasks, with build success roughly unchanged and juniors gaining most (Cui, Demirer et al.). Pick two or three pilot teams and roll out to the rest in waves.

Find the AI already in use. 84% of developers in Stack Overflow's 2025 survey used or planned to use AI tools (InfoWorld), and more than half of the 10,600 workers in BCG's 2025 survey said they'd use unapproved tools if not given approved ones (BCG, analyst). Bans push it underground: Samsung banned generative AI on company devices from May 1, 2023, after engineers pasted source code and meeting notes into ChatGPT (Fortune). Approved tools good enough to beat the shadow ones are a security control.

Write a one-page AI policy. Only 25% of US organizations had communicated a clear AI strategy in Gallup's May 2026 data (Gallup), and a clear, communicated AI stance is one of DORA's seven AI capabilities. My draft contents, built from the notes:

• Approved tools on enterprise terms: no training on your code, stated retention and region, SSO, admin audit logs, IP indemnity.
• Data classes: what may enter a prompt or agent context. Card data, production credentials, KYC files and partner-confidential material never.
• Secrets and dependencies: secret scanning before commit and in CI; lockfiles and allow-listed registries; verify any package an AI suggests.
• Agent permissions: no production write access, dev and prod separated, destructive commands need a human, each agent its own identity.
• Attribution: tag AI-assisted changes. The Linux kernel's policy (committed December 23, 2025) bars agents from certifying a contribution and asks for an Assisted-by tag, with the human submitter fully responsible (kernel.org).
• Prototype line: PM and designer prototypes live in sandboxes; promotion to production goes through normal review.
• IP: the US Copyright Office (January 29, 2025) says prompts alone don't make you an author, but human selection, arrangement or modification can (Copyright Office).

Agree the compliance position before the first agent PR. Take PCI DSS 6.2.3.1, SOC 2 CC8.1 and the EU DORA RTS Art. 17 to compliance and your assessor: who is the author when an engineer prompts an agent, what record per change they expect, and whether an AI reviewer counts as review. I found no auditor or regulator guidance specific to AI-written code, so my reading (the prompter is the author and can't be the sole approver) is a hypothesis to test with them.

Pick the scorecard now, before anyone has a result to defend. See Measuring it: adoption, flow, guardrails, value and people, reviewed monthly by you, the engineering lead and someone from compliance.

Sources: METR 2025 and paper, METR update, February 24, 2026, DORA capabilities model, Cui, Demirer et al. in Management Science.

Expect the bottleneck to move to review. In a vendor study of 10,000+ developers, high-adoption teams merged 98% more PRs while review time rose 91% and PR size 154%, with no company-level gain (Faros, July 2025, vendor study). At Google, in large AI-assisted migrations, 80% of the code in landed changes was AI-authored and total time fell by about half, but review and rollout, not generation, became the limit (Google, January 2025). DORA's 2025 survey of about 5,000 people linked AI to higher throughput and higher instability; its 2026 ROI report expects a dip before returns (InfoQ). Throughput only counts if change fail rate, rework and incidents hold.

Start agents on bounded work with an objective "done". The companies with the clearest end-to-end results picked work that CI already defines:

• Stripe's in-house agents ("Minions") open more than 1,300 PRs a week, every one reviewed by a human, submitted only after CI passes; they're best on config changes, dependency upgrades and minor refactors (Stripe, InfoQ, February-March 2026). For a regulated platform, the strongest analogue.
• Spotify's background agent had more than 1,500 merged PRs by November 2025 and saved 60-90% of time on migrations, on top of a fleet-management system that already automated about half of PRs before AI (Spotify).
• Amazon claimed in August 2024 that AI-assisted Java upgrades saved about 4,500 developer-years (The Decoder).

My first list for a platform team: dependency and CVE upgrades, migrations, flaky-test fixes, lint and config sweeps, SDK and docs regeneration. High volume, objective, low blast radius.

Strengthen the verification loop before you scale generation. Spotify's agent works because of its loops: formatters and linters as tools, tests, a model judging the diff, then human review (Spotify, part 3). Add contract tests for partner APIs and coverage on payment, messaging and identity paths. Enforce a PR size limit: DORA has long tied small batches to stability, and it's one of its seven AI capabilities.

Treat the AI reviewer as a filter, not an approver. Microsoft runs an AI pull-request reviewer on more than 90% of its PRs, over 600,000 a month (Microsoft, July 14, 2025). That's coverage, not an outcome; a regulated change still needs an accountable human. On PCI- or DORA-scoped paths, the person who prompted the change never approves it. Record per change: ticket, which tool or agent and model, who approved, test and scan results, deploy record, rollback path, and the agent's identity and permissions at the time.

Security: the code is weaker and the tools are attack surface. In a Stanford trial, people with an AI assistant wrote less secure code and were more confident it was secure (Perry et al., 2023, older models). A vendor benchmark found raw model output chose an insecure option in 45% of tasks (Veracode, vendor study). In a USENIX Security 2025 study, 19.7% of code samples referenced at least one package that didn't exist, and 43% of the fake names came back on every re-prompt, so attackers can register them first (via CSA). Commits co-authored by one coding agent leaked secrets at about twice the baseline rate (GitGuardian, vendor study). The tools themselves have shipped prompt-injection flaws; the AI agent guide explains why there's no complete fix.

Give agents no production write access. In July 2025 Replit's agent deleted a production database during a declared code freeze; Replit then separated dev and prod (AI Incident Database). The Financial Times reported a 13-hour outage of an AWS service in one China region in December 2025 after Amazon's Kiro agent chose to delete and recreate an environment; Amazon blamed misconfigured access controls, not AI (GeekWire). Both readings point the same way. The control is old; the actor is new: least privilege and separation of duties, which your audits already require.

Protect senior reviewers. Seniors carry the review tax (the 19% in Xu et al.): rotate reviewers, set review service levels and watch review load weekly.

Keep juniors learning. Juniors gain most in output, but in an Anthropic trial 52 mostly junior engineers learning a new library with AI scored 17% lower on a mastery quiz, with the biggest gap in debugging, and weren't significantly faster (Anthropic). Payroll data show employment of 22-25-year-olds in AI-exposed occupations, software included, fell about 13% relative to others from late 2022, about 16% with firm controls (Stanford Digital Economy Lab); a counter-study finds no economy-wide disruption yet. My rule: no "accept without understanding" on payments, messaging or identity code; pair AI use with debugging and code-reading exercises; keep hiring juniors, because they're your future reviewers.

Budget per merged change, not per seat. Anthropic's Claude Code docs put enterprise spend around $13 per developer per active day, $150-250 a month (docs): a planning anchor for one agentic tool. Track cost per merged change.

Don't mistake a mandate for adoption. Shopify (April 2025) made AI use a baseline expectation and part of reviews, and asked teams to show AI can't do a job before requesting headcount (Digital Commerce 360); Coinbase (August 2025) fired engineers who hadn't onboarded to AI tools within a week (TechCrunch); Amazon set an 80% weekly usage target in November 2025, then dropped its usage leaderboard after it was gamed, moving to "normalized deployments". None has published an effect on outcomes. Usage can be a team-level input metric, paired with outcome metrics from day one and kept out of individual ratings. Otherwise you're on the theatre branch.

Decide at month 3, with the scorecard, expecting a dip in the first month or two.

Sources: DORA 2025, Spotify via InfoQ, AIDev dataset on review time, Faros 2026 (vendor study), Canaries follow-up, Amazon leaderboard (secondary press).

Pick the workflow with a scorecard, not with enthusiasm. Use the seven criteria in Choosing the first workflows: volume, checkable output, decomposable, novice-heavy, written context, low emotional temperature, reversible. Every positive result in the research is high-volume.

Design for novices; leave experts room. In the support study, the AI assistant spread top performers' tacit knowledge to new agents (34% more resolutions per hour) and barely helped the most experienced. In a randomized trial of an AI assistant for Alibaba's Taobao after-sales agents, issue identification was 32% faster and ratings rose 5.3%, but objective quality didn't change and top performers had more delays and immediate customer retrials (Ni et al., 2026). On a partner-heavy platform with long onboarding (numbering rules, card disputes, cross-border corridors), that ramp time is where the money is. Don't force the assistant on your best people.

AI researches, a human decides. Stripe's compliance team splits each review into bite-sized sub-tasks; research agents fetch the evidence per sub-task; the human reviewer must answer each one; every agent action and rationale is logged for examiners. Median handling time fell 26% and reviewers rated the help above 96% (AWS and Stripe, June 26, 2026; co-published with the cloud vendor, no control group, no failure data). It turns a reviewer into an editor without moving accountability. Other published results fit the pattern: Google's security team had an AI draft incident summaries for people to accept, edit or discard and cut writing time 51% across 300 incidents (Google); Meta's root-cause tool puts the cause in its top five suggestions 42% of the time, a shortlist for engineers, not an answer (Meta). HSBC's 2-4x more true positives with 60% fewer alerts in transaction monitoring is machine-learning risk scoring, not a language model (HSBC, 2024); evidence for language models in compliance is still thin.

Route by risk and emotion before the AI touches the case. In a second Alibaba experiment, AI handled eligible chats with humans monitoring: 647 workers, 680,676 chats over 17 days in August 2024. Chats got faster, ratings on AI-handled chats fell substantially, and when frustrated customers were escalated after a bad AI exchange, human recovery mostly failed (Tuck School of Business, July 9, 2026). Escalation is not a rescue. Send angry, high-value or legally sensitive cases to people first.

Make oversight meaningful, and measure it. People agree with machines. When radiologists got wrong AI suggestions, accuracy fell from about 80% to under 20% for inexperienced readers and from 82% to 45.5% for those with 15+ years (RSNA, 2023). Developers approved about 93% of an AI coding tool's permission prompts in one lab's telemetry (from the site's AI agent research). And they lose the skill: endoscopists' detection rate in procedures without AI fell from 28.4% to 22.4% after three months with it (Budzyń et al., observational). So:

• Reviewers see the sources, can override cheaply, and are measured on accuracy, not throughput.
• Track approval rate and time per review. An approval rate near 100%, or reviews in seconds, triggers a sampling audit, because under EU and UK law it's evidence the decision is still solely automated (see Where a human must decide).
• Seed known-wrong cases into the queue each month to measure automation bias.
• Keep a no-AI path: some cases done unaided by juniors each week and reviewed by seniors.

Write the context down first. Anthropic found complex business deployments are limited by getting scattered organizational knowledge into context, not by the model (Anthropic, September 2025).

Product work: AI inside the frontier, people at the edge. Among 776 Procter & Gamble professionals, individuals with AI matched two-person teams without it (Dell'Acqua et al.). BCG consultants with AI did 12.2% more tasks, 25.1% faster, inside the "jagged frontier", but were 19 points less likely to be correct on a task outside it (Harvard Crimson). Spec drafting and synthesis are usually inside; novel judgment on thin data can be outside. Don't let synthetic users into a discovery readout: in one study, LLM "respondents" differed significantly from real survey data on 48% of coefficients and flipped the sign 32% of the time (Bisbee et al.). "Ask the data" works on curated data layers: on real enterprise schemas the best agent solved 21.3% of tasks in November 2024 (Spider 2.0; models have improved since).

Watch for gains that vanish into checking. Among 25,000 Danish workers in exposed jobs, chatbots saved about 3% of time with no measurable effect on earnings or hours, as new oversight tasks absorbed the gains (Humlum & Vestergaard, 2025).

Managers and training decide adoption. Gallup found employees whose manager supports AI use were 1.7 times as likely to use it weekly and 7.4 times as likely to say it helps them do their best work; BCG found five or more hours of training, in person with coaching, predicted regular use, while only 36% felt adequately trained (both correlational). Train your managers first, and count training in hours.

Say what happens to capacity before launch. The Commonwealth Bank of Australia reversed 45 redundancies tied to a voice bot after its union challenged the numbers (ACS Information Age, August 2025). Duolingo's CEO said an "AI-first" memo hadn't given enough context (Fortune). Tell the team whether freed time goes to backlog, redeployment or slower hiring, and involve worker representatives where they exist. In the EU, AI that allocates work to or evaluates employees is high-risk from December 2, 2027 (AI Act Annex III).

Sources: Brynjolfsson, Li & Raymond, Alibaba agentic study, Humlum & Vestergaard via NBER, Gallup, BCG (analyst), Microsoft Security Copilot trials (vendor-run).

Sequence by risk, lowest first. My default order: agent-ready developer experience (months 4-5; low exposure, and it forces API quality work); a read-only assistant over docs and account data (months 5-8); a copilot that drafts changes for a human to apply (months 8-12, after partner sign-off); autonomous actions on money, end-user messages or decisions about people only inside partner rules, with human review where the law requires it.

Get partners' sign-off criteria before you build. Partners set the gates where money and messages move:

• Card networks: Mastercard Agent Pay (April 29, 2025), Visa Intelligent Commerce (April 30, 2025) and Visa's Trusted Agent Protocol (October 14, 2025) route agent payments through tokens and registration (PYMNTS). Mastercard has published agentic commerce rules (reported January 2026); I haven't read the text, and who bears a chargeback when an agent buys the wrong thing is unsettled.
• Carriers: US business messaging needs 10DLC brand and campaign registration; T-Mobile's code of conduct fines senders, from $500 to $2,000 per violation, and blocks traffic (carrier codes summary). No carrier rule specific to AI-generated messages turned up; AI texts are simply business traffic. See the telco guide.
• Sponsor banks: accountable for customer-facing tools under the 2023 guidance, and their model-risk teams may still review generative AI even though the April 2026 US model-risk guidance excludes it (Orrick).
• Enterprise buyers: 80% of 1,169 B2B decision-makers reported stricter AI evaluations by security, legal and compliance (G2, April 2025, vendor survey).

Know the regulators' few hard lines. Disclosure: under EU AI Act Article 50, from August 2, 2026, people must be told they're interacting with AI; the Commission's July 2026 guidelines say at the very first interaction, and a generic "assistant" label or text buried in terms won't do; fines reach €15M or 3% of global turnover (Stephenson Harwood). A CPaaS or support platform is often the provider and its customer the deployer, so make disclosure the default and hard for customers to remove. The US has no federal chatbot-disclosure law, only state rules such as Utah's (FPF) and Maine's. Voice: the FCC ruled on February 8, 2024 that AI voices are "artificial" under the TCPA, so outbound AI calls need prior express consent (Cooley); Lingo Telecom paid $1M for carrying deepfake robocalls (Perkins Coie). Decisions: no AI-only adverse decisions on credit, KYC or access (see Where a human must decide).

Budget most of the effort for quality, because that's what has hurt companies. Air Canada's and Cursor's bots invented policies (see Named examples), and Klarna's CEO said a cost focus had produced lower quality (Mind the Product, May 2025). Commercial liability arrives before legal liability. So:

• Grounding: answers about policy, prices, terms and limits come only from approved sources; otherwise "I don't know" and a handoff.
• A fast human path: the US consumer-finance regulator's 2023 chatbot report warned against "doom loops" and expected a route to a human (summary); whether it still stands behind that after its 2025 guidance withdrawals is unclear, but your customers will. Measure time to a human.
• Evals from real tickets: 20-50 tasks to start, graded on the account's end state (refund issued once, rule deployed correctly), reported as reliability across repeated runs; on one benchmark, customer-service agents passed all eight tries on fewer than 25% of tasks (2024, from the site's AI agent research). Pin model snapshots and re-run evals on every model, prompt or tool change.
• Write actions: scoped short-lived credentials, approval for irreversible steps, spend caps, idempotency, a kill switch and a full audit log.

Every number in marketing needs a dated eval behind it. The FTC fined accessiBe $1M over a B2B claim that its AI made any site accessible (TechCrunch), challenged Workado's 98% accuracy claim against 53% in testing (14 News), and sued Air AI over AI marketed as replacing sales reps (DLA Piper).

Have the buyer's security answers written before sales needs them. Buyers moved from "can we opt out of AI?" in 2024 to asking you to prove their data never trains your models (SecurityPal, 2026, vendor report; no percentages). Zoom (August 2023) and Slack (May 2024) both faced backlash over terms read as allowing training on customer content (TechCrunch). The default and the opt-out mechanism are the product. Ship training off by default, an admin toggle, a public AI data-use page and a sub-processor list naming your model providers. Prepare answers to the Cloud Security Alliance's AI questionnaire (AI-CAIQ) and a view on ISO/IEC 42001 (CSA). Don't promise model-change notice longer than your labs give you: from 60 days to six months, depending on the lab.

Build developer experience for agents that actually load docs. Your customers increasingly reach your API through their own agents. Vercel tested coding agents on Next.js APIs missing from training data: 53% passed with no help, 79% with an explicitly invoked skill, and 100% with a compressed docs index in the project's AGENTS.md file; in 56% of cases the agent never invoked the skill (Vercel, January 2026, vendor study, task count not given). Next.js now bundles its docs as Markdown inside the package (Next.js 16.2). By contrast, server logs across 137,000 domains showed 97% of llms.txt files got no requests at all in May 2026 (PPC Land). llms.txt is cheap to ship; don't count it as a strategy. My agent-ready list, following Stripe's Markdown docs and official MCP server (Stripe) and Anthropic's tool guidance (Anthropic): a complete OpenAPI spec with examples and error semantics; Markdown docs, bundled with SDKs where you can; a remote MCP server with OAuth, scopes and read-only by default; a sandbox, idempotency keys and dry runs; short-lived keys for agents; and evals of common integration tasks on every docs and API release. Watch the cost: in Twilio's small test of its alpha MCP server (April 2025), tasks were faster but cost 27.5% more (Twilio, vendor test, three tasks).

Price on a unit you can audit. Intercom charges $0.99 per outcome, where an outcome includes a customer requesting no further help (Intercom); Zendesk introduced per-resolution pricing in August 2024 (Zendesk); HubSpot moved to about $0.50 per resolved conversation in April 2026 (search result). "Resolved" is the vendor's definition, and disputes move to the definition. My default: fold AI into your existing per-transaction or per-message price where it lowers your cost to serve, charge usage where it consumes compute per call, and use outcome pricing only where the outcome is objective and auditable. Put tokens, retries and the human fallback in the margin model.

Check liability and insurance before launch. If your AI screens, scores or decides for your customer, you may share liability (Mobley v. Workday); the revised EU Product Liability Directive treats software, AI included, as a product from December 9, 2026 (both from the site's AI agent research). Check your E&O and cyber cover for AI exclusions.

Sources: Visa Intelligent Commerce, FCC NPRM on AI-call disclosure, not final, Maine LD 1727, Zoom via ISACA, Stripe MCP, 2026 state chatbot laws.