My Comment Section Designed My Next Experiment. Then It Made Me Freeze My Predictions.

Security researcher Ali Afana recently conducted a rigorous experiment to test whether language models exhibit sycophancy when evaluating code vulnerabilities. Following a previous article where he observed models agreeing with scanner flags, readers challenged his methodology. In response, Afana collaborated with his audience to design a controlled experiment, incorporating a neutral arm to isolate the 'anchoring effect' of scanner warnings. Before running the tests, he publicly committed to specific predictions and statistical rules, ensuring transparency and preventing post-hoc rationalization. The results revealed that models like gpt-4o-mini act as 'over-reporters' rather than sycophants, confirming vulnerabilities regardless of external prompts. This collaborative approach highlights the value of preregistration in AI benchmarking, demonstrating how public peer review can transform anecdotal observations into disciplined scientific inquiry. The experiment underscores that model behavior is often intrinsic rather than a mere reaction to framing.