Why Does RAG Miss Information That's Clearly in the Document?

Retrieval-Augmented Generation (RAG) systems often fail to retrieve relevant information even when it exists in the source document. This article explains that the root cause is frequently poor document chunking. Because RAG relies on retrieving specific text segments to feed into an LLM, the quality of these chunks determines the system's accuracy. If chunks are too large, they lack focus; if they are too small, they lose essential context. Furthermore, arbitrary splitting can break ideas apart, preventing the model from understanding the full context. The author outlines various strategies to mitigate these issues, including fixed-size chunking with overlap, recursive structure-aware chunking, semantic chunking, parent-child chunking, and contextual chunking. Ultimately, the author emphasizes that developers should audit their chunking strategy before blaming the LLM for poor performance, as the retrieval process sets the upper limit for the model's potential accuracy.