Your AI Pilot Needs a Comparison, Not Just a Success Story

A better week is encouraging. It is not yet evidence that AI caused the improvement.

I want an AI pilot to give a business owner something more useful than a persuasive demonstration: a defensible next decision. Keep it, change it, stop it, or invest further. That decision becomes harder when the pilot is compared only with a memory of how work used to feel.

The team may have improved its intake form at the same time. The work may have become easier. An experienced employee may have taken over review. These are possible explanations, not reasons to dismiss AI. They are reasons to ask what the alternative would have produced.

Working is not the same as improving

There are three different questions. Can the workflow complete the task? Does it improve the outcome compared with a credible alternative? Does the evidence justify expanding its use? A polished demonstration can answer the first without answering the other two.

HM Treasury's impact-evaluation guidance distinguishes observing an outcome from attributing it to an intervention. Its comparison concept asks what would have happened without that intervention. I am applying that evaluation distinction to business AI, not suggesting that a small company needs a government-sized study.¹

Consider this synthetic example. A repair business uses AI to draft estimates from technician notes. Drafts arrive sooner, and the owner likes the wording. But the intake template was also redesigned, and the trial used straightforward jobs. The pilot has demonstrated useful drafting under those conditions; it has not isolated the contribution of AI or established performance on difficult jobs.

A fair alternative might be the improved template without AI. Comparing against a broken process can make a weak addition look indispensable. The relevant question is not whether AI beats your worst week; it is whether this workflow earns its place among realistic choices.

Give the comparison a fair chance

I would define the unit of work before choosing the tool: one estimate that a qualified person has reviewed, corrected where needed, and approved for the intended customer. A draft alone is a different outcome. Then I would describe the alternative precisely enough that another person could reproduce its basic conditions.

The AI-specific government guidance calls for a clearly described comparison and early baseline evidence. It distinguishes technical evaluation from impact evaluation and discusses experimental, quasi-experimental and theory-based approaches rather than prescribing one method for every situation.²

For the synthetic repair business, a possible low-risk rehearsal is to use authorized historical or synthetic job packets. Run both approaches without sending either draft to a customer. Apply the same review criteria. Where practical, conceal which process produced a draft during quality review. This is a diagnostic exercise, not automatically a representative experiment.

A live comparison needs more care. Who receives which work? Are complex jobs concentrated in one group? Do staff share the new template? Does training carry over into later work? Are reviewers applying the same standard? Ask an evaluation specialist to help when the decision, risk or investment warrants it. Random assignment is useful in appropriate designs; it is not permission to withhold needed service or experiment on people without suitable safeguards.

A before-and-after record is still useful when no stronger comparison is feasible. Describe it honestly: improvement observed during the pilot, with other explanations unresolved. That statement is less dramatic than a causal claim and more helpful to the person approving the next investment.

Check what assistance leaves behind

A recent preprint illustrates why the outcome needs careful definition. Cruces and coauthors describe a randomized online experiment with 1,174 adults aged 25 to 45. Participants completed a workplace-style problem-solving task with or without AI and then an unassisted module. The authors report larger assisted gains among participants with less education, some retained improvement afterward, and a substantial gap reappearing when assistance was removed.³

This is bounded research, not a forecast for a repair company's revenue. It does, however, make a useful distinction visible: performance with assistance and capability without it are not interchangeable outcomes.

For some workflows, continued assistance is the point. For others, you also want staff to learn, handle exceptions, or continue safely during an outage. Choose the intended outcome before interpreting the result. Do not penalize a tool for failing a goal you never designed for; do not credit it with a goal you never measured.

Write the Comparison Card before the success story

Here is the short card I would bring to a pilot review. It is my operating tool, not a validated measurement instrument.

1. Decision: what will this evidence let us keep, change, stop or expand?
2. Outcome: what completed, reviewed work counts, and what quality must remain intact?
3. Alternative: what credible process are we comparing with, including sensible non-AI improvements?
4. Allocation: how will we recognize differences in task difficulty, people, timing and learning?
5. Complete effort: where will we record review, correction, exceptions, tool expense and setup work?
6. Boundary: what would make the result inconclusive, unacceptable or unsuitable for expansion?

Keep the card short enough to use and specific enough to challenge. “Save time” is not a complete outcome. “Produce an approved estimate sooner without worsening accuracy or increasing total review effort” is closer to a decision you can inspect. The business must define its own acceptable tradeoffs; the card cannot supply them.

Do not combine every observation into one convenient score before looking at the parts. Faster drafting and slower review can coexist. Better average performance can hide an unacceptable exception. A useful comparison should expose those tensions instead of smoothing them away.

Let uncertainty shape the next move

A small pilot may show feasibility and reveal failure modes without establishing a stable business effect. Few observations, selective participants, unusual workloads and changing tools can limit what the results mean. There is no universal number of jobs that makes every pilot conclusive.

If the result is promising but uncertain, the next step can be a better comparison rather than a wider rollout. If the alternative performs just as well, keep the improvement that earned its place. If the AI workflow works only on a narrow class of tasks, that boundary may be a useful finding rather than a disappointment.

I am not arguing against ambitious AI. I am arguing for ambition that can explain its evidence. Before your next pilot begins, write down the alternative. Before it expands, ask what changed, what caused it, and what still has not been established.