TL;DR
The most rigorous evidence we have about an AI system almost always describes a system you can no longer run. A paper from September went through 11,628 studies on medical language models and found that the ones with the strongest designs tested the oldest models, and that 62 percent of the randomised trials measured a model family already discontinued. Put that next to the retirement calendar the vendors publish and the mismatch stops being an academic curiosity. It becomes a procurement problem, and healthcare has already invented a way around it that every regulated industry is about to copy.
Asking for evidence before you adopt sounds like the grown-up position. In most industries it is. With AI systems it quietly asks for something the release calendar has made impossible, and the people who look most careful end up making decisions on the oldest information in the room.
That is not a complaint about how slowly researchers work. It is arithmetic.
Six quarters of evidence, sixty days of notice
A paper posted on 10 September 2026 went through 11,628 PubMed records on medical language models, published between January 2023 and June 2026 across fourteen clinical areas. Volume grew 45-fold. The share of those studies using a randomised, controlled or prospective design: 2.5 percent.
Then comes the part that deserved more attention than it got. The gap between a model's release and the publication of studies evaluating it widened from 1.33 quarters to 6.08. Randomised trials tested models a median of 4.6 quarters older than the weaker designs. And 62 percent of those randomised trials evaluated a model family that had already been discontinued.
The better the study, the deader the model.
Now set the vendor calendar beside it. On 23 October 2026 OpenAI retired GPT-4, GPT-4 Turbo, o1, o3-mini and o4-mini, with the notice sent the previous April. Anthropic retires Claude Sonnet 4.5 on 30 November, announced on 30 September. Published notice windows run to roughly six months for generally available models, at least sixty days elsewhere, and this year at least one provider withdrew a model with no notice period at all.
Six quarters to produce the evidence. Sixty days to lose the subject of it.
The approval that measures something else
Healthcare already built a workaround, and it is worth studying because every regulated sector is about to copy it. Of the 1,614 AI-enabled devices the FDA has authorised through June 2026, roughly 96 percent came through the 510(k) pathway: you demonstrate substantial equivalence to something already on the market. That is a comparator standard, not an outcomes standard. The question it answers is whether the new thing resembles the last thing, not whether it helps the patient.
It is an honest adaptation to an impossible demand. It is also how a widely deployed sepsis prediction model reached bedsides and then, measured after deployment, returned 33 percent sensitivity and a 12 percent positive predictive value. Around seven false alarms for every alert worth acting on.
Why this matters for your business
If your procurement process asks a vendor to prove that this model works, you have written a requirement nobody can satisfy honestly, and you have handed an advantage to whoever is most comfortable implying things. Meanwhile the one artefact you actually control, your own logs, sits unexamined. (Arvind Narayanan put it bluntly this year: benchmarks tell you what a system did, logs tell you how and why.)
The fix is unglamorous. Stop validating models and start validating procedures: define the task, fix the acceptance thresholds, measure continuously against your own data, and treat the model underneath as a replaceable part. Then a retirement notice is a maintenance ticket instead of a governance crisis.
What we stopped weighing in 2019
Metrology hit this wall first and solved it. For 130 years the kilogram was a platinum cylinder near Paris, and the problem with the cylinder was that it drifted. You cannot calibrate the world against an artefact that changes. So on 20 May 2019 the kilogram was redefined in terms of the Planck constant. The object was demoted. The invariant was promoted.
David Collingridge warned in 1980 that by the time you understand a technology well enough to control it, control has become expensive. He assumed the technology would at least hold still long enough to be understood. That assumption has expired, and it took the concept of evidence with it.
We are still weighing the cylinder.