The half-life of AI benchmarks has halved: 19 months instead of 38

A researcher analyzed 224 language model benchmarks released by nine leading labs between 2018 and 2026 to determine their lifecycle. A benchmark is considered 'dead' when models reach 90% accuracy or human-level performance, or when the set is deemed obsolete. The study reveals that the half-life of benchmarks has dropped from 38 months for those released before 2023 to 19 months for newer ones. This highlights the rapid advancement of AI models, which are surpassing established limits twice as fast as before. Furthermore, the analysis found that one in seven benchmarks fails due to internal defects, a figure that rises to nearly 50% for agentic environments. The author has made the dataset and methodology publicly available, allowing the community to better evaluate the longevity of new AI testing tools.