A new study has exposed significant reliability issues in AI-generated clinical trial summaries, with leading language models producing hallucinated content that could endanger patient care.

Researchers at an unnamed institution created a benchmark framework testing three major AI models — OpenAI's GPT-4o, Anthropic's Claude Sonnet 4.6, and Google's Gemini 2.5 Flash — on their ability to accurately summarise clinical trial data for healthcare providers, patients, and insurance payers.

The framework evaluated 1,800 generated summaries across 200 clinical trials drawn from the ClinicalTrials.gov database. Each model was tested using audience-specific prompts and scored against a six-dimension faithfulness schema.

"Unsupported Claims" emerged as the dominant failure mode across all three models, with researchers recording a mean annotation score of just 1.55 out of three. This indicates that nearly half of the AI-generated content contained information not supported by the original trial data.

The study used a cross-encoder natural language inference model to score faithfulness, measuring how well the AI summaries aligned with source material without introducing fabricated details.

Knowledge graphs show promise

To address these accuracy issues, the researchers developed a knowledge-graph-augmented retrieval system that showed measurable improvements in faithfulness scores.

The augmented system produced statistically significant gains, with entailment scores improving by 0.0125 and overall faithfulness increasing by 0.0130 (p < 0.0001). However, the improvement mechanisms varied by model architecture.

GPT-4o primarily reduced contradictory statements, while Claude Sonnet 4.6 and Gemini 2.5 Flash improved through increased factual entailment with source material.

The findings highlight critical safety concerns as healthcare organisations increasingly deploy AI tools for clinical decision support. Hallucinated trial summaries could mislead treatment decisions, insurance coverage determinations, and patient understanding of therapeutic options.

The research paper, submitted to arXiv on July 10, provides the first systematic evaluation framework for clinical AI faithfulness across multiple stakeholder perspectives. The benchmark dataset and evaluation methodology are expected to inform future development of more reliable medical AI systems.