A well-formed number is not a measurement: 13 defects across 7 eval tools

A recent study highlights a critical failure mode in evaluation software for LLMs and AI agents: the tendency to produce well-formed numbers that lack valid empirical support. The author analyzed seven open-source evaluation tools, including MLflow and DSPy, identifying 13 distinct defects where tools returned results unsupported by the underlying evidence. These errors ranged from sign errors in thresholds to stale state reporting. The author emphasizes that these issues are not statistical problems but fundamental measurement failures, where software provides a precise-looking answer to the wrong question. By categorizing these defects, the report aims to improve the reliability of automated evaluation pipelines. The author, who maintains the evaluation tool Driftproof, advocates for treating evaluation software with the same skepticism as any other scientific instrument, ensuring that the code actually measures what the output claims to represent.