8 LLMs, 480 Questions, 1 Kaggle Benchmark: Who Can Explain a Traffic Drop?

Nishikanta Ray, creator of the open-source analytics platform InsightTrack, has released a comprehensive benchmark to evaluate how different Large Language Models (LLMs) perform as AI data analysts. By testing 480 scenarios—ranging from tool selection to complex traffic diagnosis—the study highlights the critical trade-offs between cost and accuracy. The benchmark reveals that while most models excel at basic tool selection, they struggle significantly with nuanced reasoning. Cheaper models often exhibit dangerous behaviors, such as inventing causes for traffic fluctuations or dismissing real issues as noise. The findings suggest that for specialized tasks like data diagnosis, reasoning capabilities are essential, and models like Gemini 3.7 Flash offer a high-performance, cost-effective balance. The project emphasizes that developers should prioritize code-based logic for threshold-heavy tasks while using LLMs primarily for interpretation, ensuring that AI assistants remain reliable tools for business decision-making.