AgentToolEval: Grading How LLM Agents Use Tools, Not Just What They Answer

Developer Abhisek Roy has introduced AgentToolEval, a new benchmarking framework designed to evaluate the reliability of LLM agents in tool-use scenarios. Moving beyond simple final-answer accuracy, the benchmark assesses the agent's decision-making process, including tool selection, argument accuracy, and efficiency. Testing eight models on Kaggle and five via Ollama, the study reveals that while frontier models like Claude Opus 5.5 and Gemini 3.5 Flash excel, smaller models can be highly effective when used specifically for decision-making tasks. The research highlights that a model's performance as an agent is heavily influenced by its architecture and intended use case. Notably, the study found that a 0.8B parameter model, 'tev1', performed strongly in decision-making despite struggling with full agent workflows. The project emphasizes the importance of grading the 'path' to an answer rather than just the result, providing a more nuanced understanding of LLM capabilities in autonomous environments.