Where Did My Data Go? Benchmarking LLMs as Data Pipeline Detectives

A new benchmark study explores the effectiveness of various Large Language Models (LLMs) in diagnosing data pipeline failures. The author simulated 15 common data engineering scenarios, ranging from ingestion errors to incorrect merges, to test whether models can accurately identify the cause of missing records or correctly report when logs are insufficient to draw a conclusion. The findings reveal that model cost and speed do not necessarily correlate with accuracy. While expensive models like Claude Sonnet 5 often guessed on ambiguous cases, cheaper alternatives like GPT-6 Luna demonstrated high reliability. The study highlights that providing a clear rulebook in the prompt is critical for preventing models from making speculative guesses on 'cold cases.' Ultimately, the research suggests that smaller, cost-effective models, when properly prompted, can serve as highly capable assistants for data engineers investigating complex pipeline issues.