I Gave ChatGPT My Full Codebase. The Results Scared Me — But Not for the Reason You Think.

A developer recently experimented by feeding their entire codebase into a large language model to assess its effectiveness as a code reviewer. While the AI proved exceptionally capable as a 'finder'—identifying forgotten dead endpoints, race conditions, and legacy configuration files—it failed significantly as a 'witness.' The model frequently hallucinated non-existent vulnerabilities with high confidence and, more dangerously, failed to flag a critical data-loss bug in a payment webhook. The author concludes that while LLMs are powerful tools for mapping architecture and surfacing potential issues, they lack the judgment required for final code verification. The primary danger lies in the model's uniform, confident tone, which makes it impossible to distinguish between genuine insights and fabricated claims. The author advises treating AI as a discovery tool rather than a final authority, emphasizing that human oversight and independent verification remain essential for secure software development.