The RAG Pipeline I Wouldn't Build the Same Way Twice

AI engineer Mithilesh Kumar explores the limitations of naive Retrieval-Augmented Generation (RAG) systems, arguing that simple vector search often fails to handle complex, real-world queries. The article details why relying solely on semantic similarity leads to precision loss, particularly with technical identifiers and multi-part questions. Kumar proposes a more robust, production-grade architecture that incorporates query routing, hybrid retrieval (combining BM25 and vector search), and cross-encoder re-ranking to improve accuracy. Furthermore, the author cautions against the premature adoption of complex agentic workflows, noting that they often introduce unnecessary latency, cost, and non-deterministic behavior. Instead, he advocates for a deterministic, layered approach where agentic loops are reserved only for complex, multi-step tasks. The piece concludes by emphasizing that data quality and independent evaluation of retrieval precision are more critical for system reliability than simply scaling model complexity.