AI chatbots reading X-rays can be dangerously confident even when they're wrong

What happened

The test ran 200 cases across 16 models and compared them against a panel of radiologists. 7 out of a possible 2,000 points. The best AI model hit 758. The scoring system rewards honesty and punishes overconfidence. Get it right with high confidence, and you earn full points. Get it wrong while claiming high confidence, and you lose a matching number.

More and more people are uploading X-rays or MRI scans to chatbots and trusting the responses. A recent study in npj Digital Medicine showed that widely used chatbots frequently give unreliable answers to medical questions. The research team accuses executives and investors of publicly overstating what AI models can do. Claims that AI systems already diagnose better than 99 percent of doctors are mostly based on anecdotes or simulations.

As recently as April, a study of 21 models that were then considered state-of-the-art showed they aren't ready for unsupervised clinical use. 0 will be expanded on a rolling basis to include new models. A full scientific publication with cost analyses and an error taxonomy has been announced. Two other recent studies on autonomous medical AI agents pointed in a different direction.

MIRA, a system for electronic health records, and AMIE were able to keep pace with general practitioners in simulated consultations. Both fueled expectations that AI could soon make diagnoses on its own. 0 authors push back: before an AI makes decisions independently, it has to know when it's better off not doing so. Then there's the problem of skill loss.

A Polish observational study from 2025 found that doctors who regularly use AI during colonoscopies detect significantly fewer precancerous lesions without the tool. 4 percent. The authors call it the "Google Maps effect": without the navigation aid, users are lost. Radiology already went through one AI hype cycle. In 2016, AI researcher Geoffrey Hinton declared that we should stop training radiologists because deep learning would soon take over the job.

Why it matters

Answer "I don't know," and you score zero but don't lose anything. A model that guesses confidently drops in the rankings even if its raw hit rate looks decent. The study tackles a point recently raised by this highly cited paper: as long as benchmarks only reward accuracy, AI models are trained to guess. In medicine, a confident misdiagnosis is far more dangerous than an honest admission of uncertainty.

There's no overall winner. Anthropic's Claude Fable 5 performed best on reliable and safe answers, leading the primary metric. Google's Gemini 3 Pro had the highest raw accuracy. 1 was the best at knowing when to hand a case off to a human. 1's hallucination rate nearly in half because the model more often refuses to answer rather than giving a wrong one. Other frontier models trend the opposite way.

5, for example, hallucinates significantly more than its predecessor because while it knows more, it's also more convinced of its wrong answers. According to the research team, several models would have scored much better if they had stayed quiet more often instead of guessing. This was especially obvious among open-weight models and those trained specifically for medical use.

They tried to answer nearly every case and were often wrong, usually with high confidence. The first version of the test painted an even starker picture. Radiologists hit 83 percent accuracy, while the best model managed only about 30 percent. Within three months, Gemini 3 Pro had already surpassed the level of resident radiologists. Accuracy is growing fast, but the models still lack any sense of their own limits.

What to watch

Colleagues like Richard Sutton agreed. Nearly ten years later, radiologists are still overburdened, and Hinton had to walk back his prediction. He had reduced the profession to image analysis and overlooked the complexity of the entire field.