Is AI ready to replace doctors yet?

A clinic visit where an artificial intelligence system greets you, listens to your symptoms, and offers a likely diagnosis may sound futuristic. Yet a large new study published in npj Digital Medicine suggests that scenario may not be far away, though medicine is not ready to take that leap just yet.

A research team from Osaka Metropolitan University, Kyoto University, Kobe University, and several medical centers across Osaka, carried out the first systematic review and meta-analysis comparing the diagnostic skills of generative AI systems with those of physicians. From more than 18,000 studies published between 2018 and 2024, they identified 83 high-quality studies spanning 20 medical specialties.

The central question: can AI really diagnose as well as a doctor?

So, what do the numbers say?

On average, AI models reached a diagnostic accuracy of 52 percent. That’s equivalent to getting just over half the answers right on a demanding medical exam. Not catastrophic, but far from reassuring when health is at stake.

When compared directly with doctors, the picture became clearer. Against non-expert physicians, such as residents or general practitioners, AI held its own. Advanced models like GPT-4, GPT-4o, Gemini, and Claude 3 even edged slightly ahead, though not by statistically meaningful margins. When set against seasoned specialists, however, human expertise easily prevailed. Experts outperformed AI by nearly 16 percentage points, showing that experience and judgment remain crucial in medical decision-making.

AI, then, can mimic junior doctors in certain contexts but cannot yet match specialists.

Although AI is not overtaking top physicians, equaling the performance of non-experts is still significant. In rural clinics or resource-limited regions, reliable diagnostic support could expand access to care. The study’s authors suggest AI could provide assistance to medical students and residents, serve as a training tool through simulated patient cases, and offer overworked doctors quick second opinions or background research. Rather than replacing physicians, AI could provide support where expertise is limited or stretched thin.

The review conducted by the authors of the paper covered a wide range of models. GPT-4 and GPT-3.5 were the most frequently tested, but Llama 2 and 3, Gemini, Claude, Perplexity, and PaLM2 were also evaluated. These systems were applied across specialties including radiology, ophthalmology, emergency medicine, and dermatology.

Dermatology stood out because visual pattern recognition plays a large role in diagnosing skin conditions, something AI handles relatively well. Still, dermatologists cautioned that images alone are not enough; real-life diagnoses also rely on patient history and context.

Should you trust an AI doctor?

The study also revealed that about 76 percent of the included studies carried a high risk of bias. One major issue is that many AI studies do not disclose the training data behind their models, making it unclear whether the systems were tested on truly unfamiliar cases. Without transparency about data sources and testing methods, confidence in AI’s clinical reliability remains limited.

The researchers call for more openness, urging developers and scientists to specify how AI models are trained and evaluated. Without that, the technology risks remaining a black box, impressive on the surface, but untrustworthy in practice.

The study portrays AI as a promising partner rather than a substitute. It resembles a bright, diligent medical student: often insightful, quick with information, but still in need of supervision from trained physicians. The researchers who conducted the study emphasize that AI should be seen as a support tool. Used carefully, it could ease workloads, enrich medical training, and extend services to patients who might otherwise lack access.

Medicine has always advanced by adopting new tools, from stethoscopes to MRI scanners. Generative AI may be the next in line. AI is improving, sometimes rivaling junior doctors, but experienced physicians remain unmatched. For now, doctors’ jobs are secure, though they may soon have a digital assistant close at hand.

If you want to learn more, read the original article titled "A systematic review and meta-analysis of diagnostic performance comparison between generative AI and physicians" on npj Digital Medicine at http://dx.doi.org/10.1038/s41746-025-01543-z.