Hospitals Are Rushing to Adopt AI — but Few Can Properly Test It

Hospitals are adopting artificial intelligence faster than they can verify it works safely. More than 90% of health systems have deployed third-party AI tools, but fewer than half have the infrastructure to properly test and validate them before clinical use.

A striking 63% describe their AI strategy as still developing or ad hoc.

Why hospital AI escapes the usual scrutiny

The instinctive assumption is that anything used in patient care has been through rigorous approval. For much clinical AI, that is not the case.

Software making a specific diagnostic claim is regulated as a medical device and must demonstrate safety and effectiveness. But a large share of hospital AI is deployed as clinical decision support — tools that flag, prioritise, predict or suggest while leaving the decision to a clinician. Much of that category sits outside device regulation, on the reasoning that a human remains in the loop.

The practical result is that validation falls to the purchaser. A hospital buying a deterioration-prediction tool is largely responsible for determining whether it works on its own patients — and fewer than half are equipped to do that.

The three specific risks

Model drift. An algorithm becomes less accurate after deployment as the world changes around it — patient populations shift, coding practices change, a new electronic record alters how data is captured. The model does not announce this; it simply becomes progressively less reliable while continuing to produce confident outputs.

Bias. Tools validated only on generic datasets carry whatever skew those datasets contained. A model trained predominantly on one population can perform measurably worse on another, and because performance degradation is uneven rather than uniform, aggregate accuracy figures can look acceptable while the tool fails for a specific group.

Absent monitoring. The most consequential gap. Traditional software either works or visibly breaks. A degrading model keeps producing plausible outputs, so without continuous monitoring nobody discovers the problem — there is no error message for being wrong more often.

Why external validation matters so much

The clearest lesson available to hospitals is that vendor performance figures frequently do not survive contact with a different institution. Models developed and validated at one health system have repeatedly performed substantially worse when independently evaluated elsewhere — not through vendor dishonesty, but because a model learns the patterns of the data it was trained on, including local documentation habits and referral patterns that do not generalise.

That is why testing against a hospital’s own population, rather than accepting vendor data, is the central discipline.

One approach that works

UPMC offers a template. It built a real-world data platform, Ahavi, to validate algorithms against de-identified patient data before deployment. It has run formal AI governance for over two years, tests vendor tools against its own patient population rather than trusting vendor data, and monitors tools after launch.

“Without…a dedicated or consistent test environment strategy, they’re going to go down that path just to learn that all that work potentially wasn’t justified,” said UPMC’s Ken Howard.

The argument in that quote is commercial rather than ethical, and it is the more persuasive one for a hospital executive. Implementing a tool that turns out not to work on your population wastes the implementation cost, the clinician training and the institutional credibility spent on it. Testing first is cheaper than discovering later.

Why most hospitals cannot do this

UPMC is a large academic health system with resources most hospitals do not have. Building a validation platform requires data infrastructure, staff who can evaluate model performance, and governance capacity to act on findings.

Time, money and talent shortages make structured testing hard to build, and the people capable of doing it are expensive and scarce. Meanwhile the pressure to adopt is real — documentation burden, staffing shortages and administrative cost are genuine problems AI vendors credibly promise to address.

A hospital declining to adopt until it can validate properly may simply be forgoing benefits its competitors are capturing.

The standards gap

There are still no consistent industry standards for assessing clinical AI, so health systems and vendors largely self-govern — and clinicians are often left to flag equity or accuracy problems manually.

Relying on clinicians as the safety net is a weak control. A clinician sees individual cases, not aggregate performance, and cannot detect that a tool has become 8% less accurate for a subgroup. Noticing a systematic failure from within a workflow is close to impossible.

Why this is a business problem too

As AI spreads into diagnosis and workflow, the gap between adoption and oversight is becoming one of healthcare’s central safety and business challenges.

The liability question alone is unresolved: when a decision-support tool contributes to a bad outcome, responsibility is distributed ambiguously between vendor, institution and clinician. Institutions deploying tools they cannot evaluate are accumulating exposure they have not measured — which may prove the argument that moves this faster than patient safety alone has. Business and health IT news.