LLMs Pass the Data-Science Quiz, Then Give Different Advice: A Kaggle Benchmark of 36 Measured Judgment Calls

A new Kaggle benchmarking study reveals a significant gap between how Large Language Models (LLMs) perform on multiple-choice data science questions versus real-world advisory tasks. By testing 11 models across 36 measured judgment calls—ranging from p-value interpretations to cross-validation strategies—the study found that while models achieve 94–100% accuracy in multiple-choice recognition, their performance drops significantly in open-ended scenarios. The research highlights that models often provide the correct practical advice but frequently rely on incorrect reasoning or flawed heuristics. The author notes that while models successfully avoid common 'traps,' they struggle to explain the underlying mechanisms of data science decisions. This benchmark, which includes models from Google, Anthropic, OpenAI, and others, suggests that while LLMs are proficient at recognizing standard solutions, they remain inconsistent in providing reliable, evidence-based guidance for complex, non-textbook data science problems.