The Platform PM

How it works

Field Guide

AI governance and model risk

02

How it works

4 min read · Chapter 2 of 10

How a use case gets approved, step by step

One AI use case, from proposal to retirement. The riskier the use, the more testing and oversight it gets before launch. After launch the model can change under you, through drift or a vendor's update, so monitoring never ends until the use case is retired.

Here's the lender's assistant, from proposal to retirement. The riskier the use, the more testing and oversight it gets before launch. After launch the model can change under you, through drift or a vendor's update, so monitoring never ends until the use case is retired.

  1. Proposed. A business owner submits the use case: purpose, users, data, which decisions it touches, which vendor and model. It gets an ID, an owner and a status in the inventory. Someone also has to find the AI already in use (SaaS features, browser extensions, personal accounts), and that list usually comes from security tools. What can go wrong: most AI never gets proposed. In a 2026 survey by OneTrust, which sells governance software, a third of companies said staff used unapproved AI because the approved route was too slow, so a slow intake creates the shadow AI it's meant to catch.
  2. Assessed. The use case is risk tiered. Is it prohibited? High-risk under the EU AI Act (credit scoring is, from December 2, 2027)? A consequential decision under a state law? How material is it, by exposure (how many decisions, how much money) and purpose? Then come impact assessments (the EU's fundamental rights assessment for credit scoring, a GDPR assessment, or ISO/IEC 42005), vendor due diligence on data use and notice of model changes, and legal review. The exception on the diagram, Blocked, happens here: the use is prohibited, such as emotion recognition at work in the EU, or too risky for the controls on offer. In a US bank there's an extra fork: under the April 2026 guidance a generative AI assistant isn't a "model", so it goes to a separate AI register. What can go wrong: the use case is under-tiered and grows quietly; the same tool counts as a model under one definition and not another; a rejected use case carries on through personal accounts.
  3. Validated. The use case is tested: task evals on real loan files with known answers, bias tests on the draft letters, red teaming for prompt injection and data leaks, and a sample of outputs checked by people. In banks this is "effective challenge", critical review by people with the expertise, independence and standing to force a change. A foundation model can't be validated like a credit score; one large bank's AI head told Risk.net "You cannot validate a foundation model", so the work shifts to testing behavior. What can go wrong: validation becomes box-ticking; the vendor won't share what proper testing needs; tests pass and nobody records which model version they ran on.
  4. Deployed. The use case is approved, usually with conditions: a human reviews each letter, volumes are capped, monitoring thresholds and a review date are set. Launch brings the notices (adverse-action reasons under Reg B, an AI disclosure under EU Article 50 for a chatbot), user training and logging; EU deployers of high-risk systems keep logs for at least six months. US bank guidance lets a model go live before validation finishes when the need is urgent, with limits and closer monitoring. What can go wrong: the conditions never get closed; the human reviewer approves everything; the conditional launch becomes permanent.
  5. Monitored. The use case is watched: quality and drift metrics, how often underwriters override the draft, complaints, incidents, cost, and the vendor's change and retirement notices. That's the diagram's note, because models change under you. In a 2023 study, GPT-4's accuracy at identifying prime numbers fell from 84% to 51% between its March and June versions; critics said that measured a shift in default answers, but the same model name behaved differently either way. Lab retirement notices run from about 60 days to six months, and every change means re-testing and sometimes re-approval. What can go wrong: the vendor retires the model and nobody re-tests; an incident isn't escalated (in the same OneTrust survey, only 27% of companies said they had slowed or paused a deployment in response to AI incidents).

Retired is the other exception, when the use case is replaced or failed. The owner tells users, switches to a fallback, archives the model version, prompts, eval results and logs, and watches for effects downstream. Canada's OSFI E-23 is the only regulator text I found that spells this out, down to keeping the retired model as a benchmark or fallback, which is impossible for a vendor model you never held. Records outlive the model: EU deployer logs, California's four-year rule for hiring data, any court order to reproduce a decision.

Cheat sheet: which rules reach which use (October 2026)

Credit decisions (US)

In force now
Reg B adverse-action reasons; state fair-lending law; FCRA notices
Coming
Colorado (January 1, 2027, may slip); California automated-decision rules (January 1, 2027)
Who carries it
The lender

Credit scoring or life and health insurance pricing (EU)

In force now
GDPR automated-decision rights; Article 50 if a chatbot faces customers
Coming
High-risk duties and a fundamental rights impact assessment (December 2, 2027)
Who carries it
Deployer: oversight, logs, assessment. Provider: conformity

Hiring and promotion

In force now
NYC bias audits; Illinois; California's rule treating vendors as agents; Connecticut's "AI is not a defense"; federal statutes through private suits
Coming
California and Colorado (January 1, 2027); Connecticut notices (October 1, 2027); EU (December 2, 2027)
Who carries it
The employer, and the vendor as its agent

Insurance pricing and claims (US)

In force now
NAIC bulletin in about 25 states; Colorado's regulation; New York guidance; state limits on AI-only prior-authorization denials
Coming
NAIC evaluation tool
Who carries it
The insurer

Customer chatbot

In force now
EU Article 50 disclosure; FTC deception rules; the Air Canada ruling
Coming
EU marking grace ends December 2, 2026
Who carries it
The company running it

Health care decision support (US)

In force now
HHS rule on discrimination in patient care tools; FDA change plans for AI devices
Coming
Who carries it
The hospital or insurer

Internal assistant, no decision about a person

In force now
Company policy, vendor terms, customer contracts; in a US bank, "broader risk management"
Coming
OSFI E-23 for Canadian banks and insurers (May 1, 2027)
Who carries it
The company