I Hid a Rule in CLAUDE.md. Only One Reviewer Could Prove It Read It.

Developer Dann Waneri conducted an experiment to determine if AI code review tools actually read repository-specific instruction files like CLAUDE.md or AGENTS.md. By creating a repository with custom, arbitrary rules—such as a specific kebab-case naming convention—and intentionally breaking them, Waneri tested the transparency of CodeRabbit and Qodo. While both tools eventually identified violations, the experiment highlighted a critical distinction: Qodo provided specific citations, pointing to the exact file, line number, and rule being violated, whereas CodeRabbit’s feedback was less transparent, making it difficult to verify if the tool was actually following the repository's rules or simply applying general coding standards. Waneri concludes that for teams scaling AI code review, the ability of a tool to provide auditable, source-linked feedback is more important than its ability to simply catch errors, as it ensures consistent enforcement of project-specific guidelines.