The Agent Finished the Job. The Benchmark Gave It Zero.

A recent study highlights the discrepancy between an AI agent's actual performance and its benchmark scores. The author developed a benchmark using 96 synthetic episodes to test how agents handle business operations, such as ticket creation and inventory management, under conditions like lost acknowledgements or timeouts. The findings reveal that while models like DeepSeek-v4-pro successfully completed business tasks, they received a zero score due to strict evaluation criteria, such as formatting errors or minor protocol violations. The author argues that relying solely on a single success flag is insufficient for evaluating autonomous agents. Instead, developers should track multiple dimensions, including business effects, authorization compliance, and recovery behavior. The study concludes that separating execution evidence from strict contract compliance is essential for building reliable agentic systems, as an application might otherwise misinterpret a formatting failure as a failed business operation, leading to dangerous duplicate actions.