Algorithm Change Protocols for Clinical AI: What to Demand

A locked algorithm produces the same output for the same input every time it runs. It changes only when someone deliberately ships a new model, and that shipment is an event you can point at. An adaptive algorithm updates itself from the data it encounters, so its behaviour drifts between releases. Almost all clinical products sold in India today are locked in this strict sense, which is a good thing, but that word carries less weight than most buyers assume.

The reason is that locked describes the model weights, not the system around them. A product can hold its weights constant and still change behaviour by moving a decision threshold, swapping a pre-processing step, changing which images are rejected as ungradable, updating a rules layer sitting on top of the model, or altering the prompt and retrieval corpus behind a language model. Every one of those changes what the clinician sees. Some vendors do not classify any of them as a model change.

So ask about the whole inference path rather than the weights. The productive question is: name every component that can be modified without my consent, and tell me what happens to the output when each one changes. Vendors who have thought about this answer immediately and often hand you a document. Vendors who have not will tell you the model is locked and try to move on to the next slide.

A change protocol is a document agreed before deployment that states what kinds of change the vendor may make, how each kind is tested, what evidence is produced, and how the hospital is informed. The idea has been formalised elsewhere, most visibly as the predetermined change control plan in United States regulatory practice, and it is worth saying plainly that this is not currently an Indian licensing pathway. You are borrowing a sound structure, not citing a rule that binds your vendor.

Build the protocol around a change taxonomy. Class one changes are cosmetic or infrastructural and cannot alter output. Class two changes alter output within a boundary the vendor commits to, such as retraining on more data of the same type with performance maintained on a fixed test set. Class three changes alter intended use, patient population, input type or output semantics. The taxonomy matters because your notice period, your revalidation duty and your right to refuse should all differ by class.

Each class then carries three obligations: a pre-release test the vendor must pass, an artefact they must hand over, and an action on the hospital side. For a class two change that might be performance on a frozen test set, a release note carrying the numbers, and a shadow comparison on your own data before anything goes live. For a class three change it should be full revalidation and a fresh approval from whichever committee signed off the original deployment.

If a clinical decision is questioned two years from now, somebody will ask what the tool said and why. Answering that requires knowing which model version was live at that moment, what input it received, what it returned, and what the clinician did next. Very few hospital deployments can reconstruct any of this, because the vendor holds the inference logs and the hospital holds only the clinical outcome and a vague memory.

Insist that the model version identifier is written into your own records at the point of use, not held solely on the vendor's side. It should sit alongside the encounter, the study or the note, inside your system, under your retention rules. Your medical records department already works to defined retention periods for inpatient records, and the model version is part of the record of how a decision was reached. HealUDoc activity logs can carry that identifier next to the encounter it influenced.

Also require that the identifier changes whenever anything in the inference path changes. A vendor who reuses a version string across a threshold adjustment has defeated the entire purpose of the exercise. And require a readable release history: date, version, class of change, what altered, and the evidence attached to it. That history is the first document an assessor or a lawyer will ask you to produce.

Vendor test-set performance tells you an update did not break the product on the population the vendor cares about. It tells you very little about what happens on your scanners, your patient mix, your typists and your languages. Revalidation on your own data is the only test that answers your question, and it is entirely reasonable to require it before a class two change is allowed anywhere near production traffic.

The practical mechanism is a shadow period. The new version runs alongside the current one on live cases, its outputs are recorded but never shown to clinicians, and you compare. The comparison you want is not accuracy in the abstract but disagreement: how often do old and new differ, in which direction, and on which kinds of case. A change that improves aggregate accuracy while flipping a patient group you care about is a change you may still be right to refuse.

This costs something real. It needs a held-out set of your own adjudicated cases, an analyst to run the comparison, and clinician time to settle disagreements. Budget it as an annual line rather than a one-off project. Hospitals that skip it end up doing the same analysis anyway, after an incident, with worse data, less time and considerably less patience in the room.

Convert all of this into contract language while you still have leverage, which means before signature and not at the first renewal. The terms that matter are notice, classification, evidence, the right to defer, rollback and continuity. Notice should be measured in weeks rather than in a changelog you have to go looking for, and it should reach a named person at the hospital rather than a shared support address that nobody reads.

The right to defer is the clause vendors resist most and the one you need most. A hospital should be able to remain on a validated version for a defined window while it revalidates, and the vendor should be obliged to support that version throughout. Without it, an announcement that everyone has been updated arrives as a completed fact on a Monday morning. Expect to concede something here, usually a maximum deferral period and a security exception allowing urgent fixes to ship immediately.

Rollback is the safety net underneath all of it. If a new version behaves badly in your setting, you need a defined path back to the previous one within a stated time, and the vendor needs to have retained that version in a deployable state. Cloud-delivered products often cannot do this at all because they run a single version for every customer. Establish that before signing rather than during the incident review.

A change protocol that nobody executes is worse than having none, because it creates a documented obligation you visibly ignored. Assign it properly: someone owns receiving vendor notices, someone owns the revalidation run, and the AI committee owns the decision to accept or defer. In most hospitals the natural home is clinical informatics with the quality department as a standing reviewer, but the point is that it is named rather than that it sits in any particular office.

Keep a register of deployed versions per tool, with the date each went live and the date it was retired. Cross-check that register against the vendor's own release history every quarter. Discrepancies are informative rather than embarrassing: they usually mean either that the vendor pushed something without notice, or that your environment is quietly running an older build than everybody believes it is.

Finally, treat the protocol as reciprocal. You will change too: a new scanner, a replaced LIS, a renamed charge code, a different consultant mix, a redesigned documentation template. Any of these can shift the input distribution the model sees, and none of them will look like an AI change to the person making it. The vendor should be told about those changes as formally as you expect to be told about theirs.