Should This AI Pilot Go Live? The Production Gate for Billing and Customer Operations
There is a specific moment in every AI programme where the interesting question stops being does this work and becomes is this safe to operate.
It arrives when someone proposes that a model influence billing, collections, customer service, move-in and move-out workflows, or anything else a customer will feel. Up to that point the pilot has been graded on capability. From that point it has to be graded on consequence, and the two are not the same evaluation.
This is a gate, not a review. It produces one of four outcomes — scale, redesign, pause, or retire — and the third and fourth are legitimate results, not failures of nerve.
One distinction before anything else, because it determines how much of this applies. An AI tool that helps an analyst search internal documents is a productivity question; if it's wrong, someone notices and moves on. An AI system that recommends, drafts, routes, updates or triggers something in a customer-facing workflow is a different class of object. It can produce an incorrect bill, miss a regulatory obligation, create a queue nobody has capacity to clear, or tell a customer something the organisation is then bound by. Everything below is aimed at the second kind.
I've written separately about the five properties a system needs before it acts without review. This is the operational version of that argument: what you actually measure, and what stops the launch.
Why pilots fail after passing a controlled test
A pilot is an experiment conducted under conditions you selected. Production is conducted under conditions that select you.
The data is not the data you tested on. Pilot datasets are curated by definition — someone chose a clean period, a representative segment, accounts without the awkward history. Production brings stale meter reads, accounts with partial histories, consent records that exist in a different system or not at all, tariff data that disagrees between sources, and documents whose provenance nobody can establish. Each of these degrades output quality in ways offline evaluation cannot surface, because the offline set didn't contain them.
The integration surface is larger than the model. A billing-adjacent AI system touches the billing engine, the CRM, the customer portal, payment systems, document stores and workflow tools. Each is a source of latency, a source of stale state, and a place where an assumption about data shape turns out to be wrong under load.
Non-quality dimensions block launches that quality would have passed. Access control, customer-data segmentation, retention obligations, explainability requirements, p95 latency under real concurrency, and whether the people meant to use it actually will. A system can be accurate and still be un-launchable on any one of these.
Plausible is not the same as supported. This is the failure mode most specific to language models in operational settings. An answer can be fluent, correct-sounding, consistent with the account in question — and unable to point at the source record that justifies it. In a workflow where a customer may later dispute what they were told, an unsupported answer is a liability regardless of whether it happened to be right. Grounded and accurate are separate measurements and only one of them is usually being tracked.
Human review is the bottleneck nobody models. Most pilots pass their gate by promising human oversight, then size the review queue from pilot volume. At production volume the queue either exceeds capacity — so exceptions age and customers wait — or reviewers adapt by approving without reading, which converts oversight into a signature. Both outcomes are worse than the honest version, which is to reduce scope until the queue fits the staffing.
| Pilot conditions | Production conditions | |
|---|---|---|
| Data | Curated, recent, complete | Stale, partial, contradictory across systems |
| Volume | Bounded, known | Variable, seasonal, spiky |
| Integration | Often mocked or read-only | Live systems with real state and latency |
| Failure cost | An observation in a report | A customer-visible event with a paper trail |
| Oversight | Attentive, unpaid attention | A queue with finite staffed capacity |
| Success metric | Offline accuracy | Grounded, supported, reversible, staffed |
The production-readiness scorecard
Score ten categories, weighted to the risk tier of the workflow. Every score requires evidence — a test result, a log extract, a signed control, a measured number. Stakeholder confidence is not evidence, and neither is a demo.
| Category | What it evidences |
|---|---|
| Value | Verified benefit net of full cost, with stated assumptions |
| Output quality | Grounded-answer rate, task completion, incorrect-action rate |
| Safety | Prohibited-action rate, policy-violation rate, escalation accuracy |
| Security & privacy | Access control, segmentation, secrets handling, retention |
| Data readiness | Freshness, completeness, source traceability, consent records |
| Integration readiness | Latency under load, error handling, state consistency |
| Operability | Monitoring, alerting, rollback, incident path, staffed review |
| Auditability | Decision records that resolve to real source identifiers |
| Adoption | Correct workflow use and override rate, not logins |
| Ownership | A named accountable business owner, agreed before launch |
The total is advisory. The no-go conditions are not.
Any of the following blocks launch regardless of the weighted score:
- A decision cannot be reconstructed after the fact
- Customer data is reachable outside its intended segment
- A required regulatory obligation is unmet or unassessed
- No rollback exists for an action the system can take
- A single wrong action has unbounded customer or financial impact
- No named owner has accepted accountability for outcomes
A high score with an unmet no-go condition is not a close call. It is a no, and treating it as a trade-off is how organisations talk themselves into incidents.
Thresholds must be calibrated per workflow. A drafting assistant with mandatory human approval and an autonomous adjustment posting are not the same risk tier, and copying thresholds between them is the most common way a gate becomes theatre.
Defining the thresholds
The metrics matter less than the fact that they are defined, measured on production-like data, and agreed before the result is known. Setting a threshold after seeing the number is not a threshold.
| Dimension | Metric | Release implication |
|---|---|---|
| Quality | Grounded-answer rate — share of outputs citing a valid, resolvable source record | Below threshold → read-only or blocked |
| Quality | Incorrect-action rate on a held-out production-like set | Below threshold → redesign |
| Safety | Prohibited-action rate — actions outside the permitted set | Non-zero → blocked, no exceptions |
| Safety | Escalation accuracy — share of escalations that should have escalated | Very high → escalating too little. Very low → rubber stamp. |
| Operational | p95 latency at projected peak concurrency | Above threshold → capacity work before launch |
| Operational | Data freshness at time of decision | Stale beyond tolerance → fail closed, not open |
| Operational | Exception backlog against staffed review capacity | Exceeds capacity → narrow scope |
| Financial | Cost per completed transaction, including review labour | Above avoided cost → no value case |
| Financial | Expected error cost — error rate × average remediation cost | Compare against benefit, not against zero |
| Adoption | Override rate and workflow-correct usage | Very high override → the system isn't trusted or isn't right |
Two of these are routinely omitted and both are load-bearing. Expected error cost belongs in the value calculation, not in a risk register nobody prices. And review labour is a real cost of the AI system — a workflow that shifts effort from doing to checking has not necessarily saved anything.
Worked example: a billing-explanation assistant
An assistant that drafts responses for customer-service agents explaining why a bill is what it is, grounded in approved account, tariff and policy sources. The agent reviews and sends. The system never contacts a customer directly and cannot post an adjustment.
That scoping is doing most of the safety work, and it's the first thing to decide rather than the last.
Value, stated as a fraction rather than asserted as ROI:
Numerator — verified benefit
- Reduced handle time per contact × contact volume
- Avoided rework where the first explanation was wrong
- Prevented escalations to a supervisor or complaints process
- Avoided erroneous goodwill credits issued to close an argument the agent couldn't win with facts
Denominator — full cost
- Model usage and hosting
- Integration and data-preparation engineering
- Ongoing support and on-call
- Security and privacy review
- Training and change management
- Human review labour at production volume
- Incident handling and remediation
Variables that decide the answer
- Expected monthly volume
- Review rate — share of outputs a human must check
- Exception rate — share that can't be handled and must route elsewhere
- Unsupported-answer rate — outputs with no resolvable source
- Average error cost
- Confidence threshold, which trades review rate against exception rate
- Average human review handling time
Publish which assumptions would flip the recommendation. For a workflow like this, the value case is usually most sensitive to review rate and error cost — and both are systematically underestimated at pilot stage, because pilot reviewers are motivated and pilot errors are caught by people who were looking for them.
A calculation with stated inputs that a reader can disagree with is worth more than a ROI figure they cannot check.
The controls around the model matter more than the model
A reference shape for an AI system that supports billing or customer operations, with the control points numbered where they belong:
- Source systems — billing engine, CRM, tariff catalogue, document store, payment platform
- Data pipeline with validation — freshness checks, completeness checks, schema enforcement. Fail closed: a decision made on stale data should not be made.
- Retrieval or feature layer — the boundary where "what does the system know" becomes answerable and auditable
- Model integration — retrieval-augmented for policy and account questions; predictive models for scored decisions; constrained agent patterns where a sequence of steps is genuinely needed
- Policy and tool constraints — the permitted action set, enforced in code rather than requested in a prompt
- Application and workflow orchestration — approval steps, escalation routes, exception handling
- Interface — where the human sees the output, the confidence, and the sources
- Audit log — spanning all of the above, not bolted to the end
Two constraints are worth stating flatly.
No uncontrolled write access to the billing platform. Whether the system of record is SAP IS-U, powercloud, or something built in-house, an AI layer should reach it through a constrained interface with an explicit permitted-operation set — not with general credentials and good intentions. This matters most during migration programmes, when the temptation to let a new layer write directly into a legacy platform is highest and the rollback story is worst.
Retrieval is an audit boundary. If you cannot enumerate what the system saw when it produced a given output, you cannot explain that output later. Log the retrieved source identifiers, not a similarity score.
Deployment topology — private, public, or multi-cloud — follows from data residency and existing obligations. No single answer is universally correct, and vendors claiming otherwise are describing their own architecture.
Operationalise before you launch
Production readiness is a continuous property, not a checklist you complete once.
Version everything that changes behaviour: prompts, models, retrieval configuration, policy documents, workflow definitions. A system where the prompt changed but nothing recorded that it changed cannot be debugged, and cannot be explained to anyone who asks why last month's answers differed.
Test before release: a maintained evaluation set drawn from production-like data, regression tests against known-hard cases, adversarial testing against the prohibited-action list, and a recorded approval.
Monitor in production for: drift · source freshness · unsupported-answer rate · tool-call failures · human override rate · exception backlog and age · p95 latency · cost per transaction · customer-impact incidents.
Override rate deserves particular attention. It is the cheapest available signal of whether the system is trusted, and a sharp move in either direction means something changed that nobody logged.
Release through stages, not events: shadow mode first — the system runs and records but affects nothing, which is the only way to measure production behaviour without production risk. Then canary on a limited segment behind a feature flag. Then staged expansion against monitored thresholds.
Agree rollback criteria before launch, in writing. Which metric, at which value, for how long, triggers what — decided while everyone is calm. A kill switch nobody has agreed the conditions for is a kill switch that gets debated during an incident.
Have an incident path for customer harm, data exposure, incorrect billing guidance, service degradation, and model-provider outage. The last one is not hypothetical and is not in your control.
The log fields that reveal risk
What a production record needs to contain to be worth having:
| Field | What it detects |
|---|---|
| Workflow and request ID | Traceability end to end |
| Model, prompt and policy version | Whether a behaviour change had a cause |
| Retrieved source record IDs | Whether the output was grounded in anything |
| Data freshness at decision time | Decisions made on stale state |
| Confidence score | Threshold calibration |
| Answer-support status | Unsupported outputs reaching users |
| Tool calls attempted and permitted | Attempts outside the allowed set |
| Human override and reason | Trust, and where the system is wrong |
| Exception status and age | Review queue health |
| Latency and cost | Capacity and unit economics |
| Final resolution | Whether the case actually closed |
Each field needs a threshold that triggers investigation and a named team that responds. A dashboard nobody owns is a dashboard nobody reads.
Who is allowed to approve this
Governance embedded in delivery gates works. Governance as a final compliance review does not, because by then the cost of saying no is political rather than technical.
| Decision | Typically accountable |
|---|---|
| Pilot charter and scope | Business process owner |
| Data access and lawful basis | Data protection / privacy |
| Model evaluation results | Engineering with independent review |
| Security controls sign-off | Security |
| Regulatory assessment | Compliance / legal |
| Launch decision | Named business owner |
| Incident ownership | Operations, with a defined escalation |
| Customer communication | Customer operations |
| Periodic recertification | Business owner, on a fixed cadence |
The row that matters is the launch decision, and it must resolve to a person, not a committee. A committee cannot be accountable for a customer outcome; it can only distribute the feeling of accountability until none of it lands anywhere.
Match autonomy to risk tier. Read-only assistance where outputs inform a human. Human approval before action where the system proposes and a person commits. Constrained automation where the action set is narrow, bounded and reversible. Prohibited where the consequence is severe, irreversible, or legally reserved to a human decision-maker — a category that should exist and be populated.
A note on the regulatory clock. As of 2 August 2026 the EU AI Act's Article 50 transparency obligations and the Commission's general-purpose AI enforcement powers apply. The high-risk obligations, meanwhile, were deferred through the simplification package — Annex III standalone systems to December 2027, Annex I embedded systems to August 2028. It would be easy to read the deferral as breathing room. It isn't, in the only sense that matters here: the deadline moved and the gate did not. Your auditors, your customers and your own risk function ask the same questions on the same schedule they always did. The regulation named them; it did not create them.
Verify current dates against the Official Journal text before relying on any of this — the amendment is recent and I am not a lawyer.
From one pilot to a portfolio
The staged path: use-case selection → pilot validation → shadow mode → limited production → operationalisation → scaled adoption. Skipping shadow mode is the most common shortcut and the most expensive, because it is the only stage that measures real behaviour at zero customer risk.
Prioritise across candidates on business value, data readiness, regulatory risk, workflow criticality, time to value, and whether the work builds reusable platform capability. The last is systematically undervalued: the second and third use cases are cheap only if the first one built retrieval, audit logging and evaluation infrastructure that survives.
Measure adoption by changed decisions and improved workflow outcomes. Logins measure curiosity.
And expect resistance where the system changes what someone's job is, because that resistance is usually well-founded. The people asked to review AI output are being handed accountability for a system they did not design and cannot inspect. Involve them early, make the escalation route real, and be honest about how their performance will be measured now that part of the work is done by something else. Teams that skip this discover that the humans in "human in the loop" have quietly stopped looking.
The instruction that this all reduces to: do not promote a pilot because it demos well. Release it when the evidence clears the gate, and be willing to record that it didn't.
Common questions
How do you move an AI pilot to production? Through a gate that scores value, quality, safety, security, data readiness, integration, operability, auditability, adoption and ownership against pre-agreed thresholds — then a staged release through shadow mode, canary and limited production with rollback criteria agreed in advance.
Why do AI pilots fail to reach production? Rarely because of model accuracy. Usually because production data is worse than pilot data, the integration surface is larger than expected, human review capacity doesn't scale with volume, or a decision cannot be reconstructed after the fact.
What makes an AI system production-ready for billing? Decisions traceable to resolvable source records, a permitted action set enforced in code, a rollback path for every action, escalation thresholds derived from measurement, and a named accountable owner.
Who is responsible when an AI system makes a mistake? Whoever was named before launch. If that was not decided in advance, it will be decided during the incident, by proximity.
Does the EU AI Act high-risk deferral mean this can wait? The regulatory deadline moved for certain categories. The operational gate did not — it is imposed by auditors, customers and internal risk functions independently of the statute.
I write about autonomous systems that act, and what it takes to let them. More writing here, or subscribe by RSS.