The pilot succeeded and nothing happened
The pattern is familiar to anyone who has run one. A vendor demonstrates a capability, a department agrees to a pilot, the pilot runs for a few months on a subset of cases, and the report concludes that the tool performed well. Then the pilot ends, the tool is switched off, and the hospital is exactly where it started apart from having spent the time.
What is striking is that the pilots usually do work in the narrow sense. The model performs, the clinicians who used it are broadly positive, and the report is honest. The failure is that a pilot answers the question of whether the tool works and never touches the questions that actually determine deployment: whether it can be integrated, who will own it, who pays for it, what happens when it is wrong, and whether the workflow it assumes is the workflow the hospital has.
Those questions are all answerable before a pilot starts, and answering them first changes what you pilot and often whether you pilot at all. A hospital that establishes it has no route to integrate a tool has saved itself three months by finding out in a meeting rather than at the end of a trial.
Integration is the commonest hard blocker
Almost every clinical AI tool needs data in and results out, and in a pilot both are usually arranged manually — a data extract prepared by hand, results reviewed in the vendor's own interface. That arrangement is fine for a trial and impossible in production, and the gap between them is frequently larger than anyone estimated.
Establish before piloting what the production integration would actually be: which system provides the input, in what form, at what frequency, and where the output has to appear for a clinician to act on it without leaving their normal workflow. A result that requires a clinician to open a second application is a result most clinicians will not look at, however good it is.
Then establish whether that integration is possible with your systems and your vendor contracts, and what it costs. This is where many pilots die silently: the integration is technically feasible, requires work from an incumbent vendor, and that vendor quotes a price or a timeline that ends the discussion. Finding that out first is worth a phone call before it is worth a pilot.
Integration questions to settle before the pilot
- Which system supplies the input, in what format and at what frequency
- Where the output must appear for a clinician to act on it in workflow
- Whether your incumbent vendor will build it, at what cost and when
- How patient identity is matched between the systems
- What happens to the workflow when the integration is unavailable
Workflow fit, which pilots systematically flatter
Pilots run in favourable conditions almost by definition. An enthusiastic clinician, a selected subset of cases, extra attention from the vendor, and participants who know they are in a trial. Production is the opposite: everyone, all cases, no special attention, and no awareness that anything is being evaluated.
The question worth asking is what the tool requires of the person using it and whether that is sustainable at full volume on a bad day. A tool that adds thirty seconds per case is negligible in a pilot on twenty cases and significant across a full clinic list. A tool that requires a clinician to review and confirm an output is only useful if the review is genuinely performed, and review quality degrades sharply when the tool is usually right.
Design the pilot to test this rather than to avoid it. Run it on unselected consecutive cases where possible, include the clinicians who are least enthusiastic rather than only the champion, and measure the time cost honestly. A pilot that only involves people who wanted it tells you nothing about whether the department will use it.
Ownership after the vendor leaves
Every deployed AI tool needs an owner, and the absence of one is why tools that were successfully deployed quietly stop being used a year later. Ownership here means someone responsible for monitoring that it is still working, for handling the cases where it is wrong, for deciding whether a vendor model update is acceptable, and for the periodic review of whether it should still be running.
That is a real ongoing commitment and it usually has no obvious home. It is not quite IT, not quite the clinical department, and not quite quality. Hospitals that deploy successfully name the owner before deployment and give them the time; hospitals that do not find that everyone assumed someone else was watching.
Ask what the operating burden looks like at full volume before committing: who handles a disagreement between the tool and a clinician, who is called when it produces obviously wrong output at midnight, who reviews performance monthly, and what the escalation is to the vendor. If those answers do not exist, the deployment is not ready regardless of how the pilot went.
Ownership questions with named answers required
- Who monitors that the tool is running and performing as expected
- Who handles disagreement between the tool and a clinician
- Who decides whether a vendor model update is accepted
- Who reviews performance on a stated cycle and reports it where
- What triggers a decision to switch it off
“The pilot was a success and we deployed it. Eighteen months later I discovered it had been silently failing for five months. Nobody owned it, so nobody noticed, and the clinicians had simply stopped mentioning it.”
Data quality, which is usually the real constraint
Models depend on the data available to them, and hospital data is messier than pilots suggest because pilot datasets are cleaned. Fields that are inconsistently populated, coding that varies between clinicians, free text where structure was assumed, and timestamps that record when something was entered rather than when it happened are all ordinary and all degrade model performance in production.
Assess this before piloting by examining the actual fields the tool depends on across a real period: how complete are they, how consistent, and how do they behave at nights and weekends when documentation is thinnest. A tool depending on a field that is populated in seventy per cent of cases will behave very differently from what a pilot on complete records suggested.
Where the data is not good enough, that is a finding worth having and is frequently more valuable than the tool would have been. Improving the completeness and structure of a small number of important fields benefits far more than one AI use case, and it is the prerequisite for most of them. Hospitals that discover this and act on it are better positioned two years later than hospitals that deployed something on top of poor data.
Deciding the success criteria before you start
Pilots without pre-stated success criteria always succeed, because the criteria are inferred from the results. Agreeing in advance what would constitute enough to deploy, and what would constitute a reason not to, is the discipline that makes a pilot a decision rather than a demonstration.
State the criteria in terms of the clinical or operational outcome you want rather than model performance alone. A model that is highly accurate but does not change any decision has not earned deployment. The question is what would be different — a decision made earlier, a case identified that would have been missed, time released, an error prevented — and whether the pilot showed that.
Include the cost side in the criteria: the licence, the integration, the operating burden and the clinician time. A tool that delivers a real benefit at a cost the hospital cannot sustain is a no, and that is a legitimate pilot outcome that pilots almost never state because nobody agreed the threshold beforehand.
A readiness checklist worth running first
Before committing to any pilot, run through the same short set of questions. Is there a defined clinical or operational problem, owned by someone who wants it solved. Is the integration route known, possible and costed. Are the data fields the tool depends on adequate in your real records. Is there a named owner for production operation. Are the success criteria agreed and written down, including cost. Is there a decision-maker who has committed to act on the result either way.
A proposal that cannot answer those is not ready to pilot, and saying so is not obstruction. It is the difference between an organisation that accumulates pilots and one that accumulates capability. Most stalled pilots in most hospitals failed one of those questions at the outset and nobody asked.
Keep the answers and the eventual outcome in the same governance record as your other use-case decisions, so the organisation learns across pilots rather than repeating them. A hospital that can see why its last four pilots did not deploy will design the fifth differently, which is the only route to getting good at this.
