Same Patient, Conflicting Documents: Can AI Preserve the Evidence?

A recent benchmarking experiment, ChartReplay, explores the challenges of using Large Language Models (LLMs) to maintain accurate, source-grounded medical records. The study highlights a critical issue: when AI models extract data from multiple, potentially conflicting documents, they often omit assertions or fail to preserve the provenance of information. Testing models like GPT-6 and Claude Sonnet 5, the research demonstrates that even when a final record appears correct, the underlying history may be incomplete, effectively hiding unresolved conflicts or superseded data. The author argues that for AI to be reliable in clinical settings, systems must maintain a clear, inspectable relationship between the final record and its source documents. The findings suggest that while full reconstruction of records can improve accuracy, the current 'extract-plus-ledger' approach often struggles to maintain the integrity of the patient's longitudinal history, posing significant risks for data-driven medical decision-making.