Roni Rechter

Nobody Is Reviewing the Output

Almost everything written about production AI is about accuracy. Evaluation harnesses, hallucination rates, benchmark scores, context windows, retrieval quality. It is a reasonable obsession, and it rests on an assumption so common that it usually goes unstated: that a person reads the output before it matters.

In most deployed systems, one does. The model drafts the email and you send it. It suggests the code and you merge it. It summarises the document and you decide what to do. The human is the last gate, and in that architecture accuracy really is the whole game — because the cost of being wrong is bounded by one person's attention for one minute.

Take the human out and the cost structure does not degrade gracefully. It changes category.

An agent that issues an invoice has issued it. An agent that moves money has moved it. By the time anyone could object, the action is in the world, in someone else's bank account, in a regulatory record. The question stops being was it right and becomes a set of much harder ones:

Not one of those is a model question. They are systems-engineering questions, and they are most of the distance between a demo that impresses a steering committee and something allowed to touch real money.

I build billing systems. That means I work in an environment where a wrong output is not a bad suggestion — it is a compliance event with a paper trail, a customer who noticed, and a regulator entitled to ask why. It is an unglamorous place to work on autonomy and an unusually instructive one, because every shortcut gets found.

Here is what I have come to think a system needs before it is allowed to act without review. Five properties. None of them are about the model.

1. A decision record, in a vocabulary that still resolves

Logs tell you what happened. A decision record tells you why: the inputs as they were at the time, the model and prompt version, which deterministic rule fired, what the confidence was, and what alternative the system rejected.

But capturing enough fields is not sufficient, and this is where we learned something.

One of our systems generates commercial offers — catalogues and pricing models — with competitor catalogue data available to reason against. The outputs were, semantically, fine. Operationally they were inert, because the model named things its own way. Nothing downstream could match its output to anything: not to our predefined catalogue entries, not even to the competitor data it had ostensibly been comparing against.

So when the question came — why did it propose this — what we had was a faithful record of the model's own private vocabulary. Every field was present. None of it joined to anything.

The fix was to make the generated catalogue speak the predefined naming, so that output lands in the same terms as the reference data. But the fix is not the interesting part. The interesting part is that a decision record expressed in a vocabulary only the model uses is not a decision record. It is an artefact that looks like accountability and cannot answer a question.

Any agent emitting structured output needs a controlled vocabulary, imposed from outside, or its own history becomes unqueryable. You will not notice while it is working. You will notice the first time you need to explain a specific decision from eight months ago.

2. A blast radius bounded in code

Every autonomous action has a maximum plausible damage. The property that matters is whether that maximum is enforced structurally or merely requested.

Prompts are not controls. An instruction the model could reason its way around is not a limit; it is a preference. But there is a stronger version of this, and it took a while to see clearly.

Language models cannot do arithmetic reliably. That much is common knowledge, and the usual conclusion is to keep them away from numbers. The more useful conclusion is that they do not need to calculate — they can write a program that calculates.

So in our pricing work the model does not compute a tariff. It generates a script that computes it. Which relocates the control problem entirely: the prompt was never where the constraint belonged, because the prompt is not where the work happens. Careful instructions about numerical rigour are addressed to something that isn't doing the maths.

The constraint belongs in two places. On the generated script — what it may import and call, where it runs, whether it passes static inspection before it ever executes. And on the script's output — bounds, sanity checks, reconciliation against values already known to be true.

This generalises further than pricing. Having a model emit code rather than answers converts a probabilistic step into a deterministic, inspectable artefact, and deterministic artefacts can be constrained in ways model outputs cannot. The blast radius stops being whatever the model says and becomes whatever this specific reviewable program is permitted to do. That is a much better place to be standing, and it is available more often than people use it.

3. A reversal path with a known window

The rule we work to is that anything with a lifecycle, and anything that gets updated, carries an audit trail and version history from the start.

That reads like a governance checkbox. It is actually the reversal mechanism — you can only return to a prior state somewhere you kept one.

Which makes reversibility a design-time decision rather than an incident-time capability. Choosing which entities have lifecycles is the same act as choosing what can be undone. Anything modelled as a fire-and-forget event — emitted, never versioned — is unreversible by construction, and that gets discovered during the incident rather than before it.

Worth separating two questions that often get merged: can this be undone, and does undoing it create a second event? A reversed invoice is not a return to the prior state. It is a new customer-visible fact, possibly a new accounting entry, possibly a new disclosure. Reversal is a compensating action, not an eraser.

4. Calibrated escalation

Every autonomous system needs a way to stop and ask. Almost all of them get the threshold wrong in one of two directions.

Set it too high and the system never escalates, which means uncertainty is invisible right up until it is expensive. Set it too low and it escalates constantly, humans learn the queue is noise, and within a fortnight they are approving without reading — which is worse than no escalation at all, because now there is a signature on it.

The threshold is an engineering parameter and it should come from measurement rather than instinct: what fraction of escalations turn out to have been correct to escalate? If that number is very high, you are escalating too little. If it is very low, you have built a rubber stamp.

5. Liability assigned before the incident

Someone must be accountable for what the system does, named in advance.

This sounds procedural and it is the property most often skipped, because it is the only one that cannot be solved by writing code. If the answer to "who is responsible when this is wrong" gets determined during the incident, it was not determined — it was assigned to whoever was in the room. Organisations discover this at the worst possible moment, and the discovery is usually that everyone assumed it was someone else.

In a regulated context there is frequently a legal answer whether you have chosen one or not, which is a strong argument for choosing deliberately.

Why this is the gap

Most enterprise AI projects do not fail because the model was insufficiently accurate. They fail somewhere between a successful pilot and production, and they fail because none of the five properties above were in scope. The pilot demonstrated capability. The gate that follows asks for accountability, and capability is not evidence of it.

That gate is not bureaucratic obstruction, though it is often experienced that way. It is the correct question, asked by people whose job is to ask it, about a system that is about to do something irreversible to a real customer.

The teams that get through do not have better models. They have answers.

Where I think this is incomplete

Two honest weaknesses.

The first is that these properties are much easier to state than to price. Every one of them is real engineering effort with no demo value, which makes them structurally hard to fund. I do not have a clean answer for how you justify the cost before an incident makes it self-evident.

The second is that I worked this out in a domain — regulated financial operations — where the constraints are unusually legible. There is a written rule, an auditor, and a defined consequence. I suspect the same five properties matter in settings with no regulator, and I suspect they are much harder to argue for there, because nothing external forces the question.

I write about autonomous systems that act, and what it takes to let them. More writing here, or subscribe by RSS.