Your tool returned the rows. The model counted them wrong.

A recent investigation into AI agent frameworks reveals a critical reliability issue: LLMs often fail to accurately count items when provided with raw data lists. By testing 408 runs across frameworks like CrewAI, LangGraph, and Strands, the author demonstrates that models frequently produce off-by-one errors or misinterpret boundary conditions when tasked with counting rows returned by a tool. The study highlights that these failures occur silently, without triggering errors, and are often caused by the model's inability to process large data lists effectively. The author suggests that developers should precompute counts within the tool itself rather than relying on the model to perform arithmetic on raw data. This approach not only improves accuracy but also significantly reduces token usage and costs. The findings emphasize the importance of verifying exactly what data reaches the model's prompt versus what the tool initially outputs.