I Let AI Write My Tests for 6 Months. Here Is What Actually Survived Production

A software developer shares their six-month experience using AI to generate automated tests, highlighting both its utility and significant limitations. While AI excels at creating boilerplate code, handling bug reports, and suggesting stable locators, it often fails to understand domain-specific logic or edge cases. The author notes that AI struggles particularly with mobile web and Flutter environments, often producing outdated or incorrect code. The article emphasizes that while AI can accelerate the creation of test skeletons, it cannot replace the human ability to reason about failure modes. The author concludes that a successful workflow involves using AI for scaffolding while manually writing all assertions to ensure true quality. Relying solely on AI-generated tests risks creating a false sense of security, as these tests often check for existence rather than correctness, ultimately failing to catch critical production bugs.