Drafts That Never Shipped: Do LLMs Treat Dead Proposals as Real Standards?

A new benchmark study explores whether Large Language Models (LLMs) mistakenly treat rejected or abandoned technical proposals—such as withdrawn emoji, deferred Python PEPs, and expired IETF drafts—as established standards. The research tested eight models from five different labs, including frontier models like Claude Sonnet 5 and Gemini 3.7 Flash. Findings reveal that models frequently hallucinate these 'dead' proposals as active, with 26% of answers asserting that non-shipped standards are real. Even when models correctly identify a proposal's status, they often fail to exclude it from generated code or citations, leading to non-compiling syntax or fake RFC references. The study highlights that while providing ground-truth evidence in prompts significantly reduces these errors, the issue remains a persistent challenge across both small and large models, suggesting that training data containing outdated specs is a primary source of these persistent technical hallucinations.