I described 1,245 tables with an LLM and retrieval got worse

In a recent technical deep dive, developer Ashish Sinha explores the counterintuitive failure of using LLMs to generate descriptive metadata for database schemas. While cataloging tables with AI-generated summaries is intended to improve search retrieval, Sinha found that it significantly degraded performance. The issue stems from two mechanisms: IDF (Inverse Document Frequency) collapse, where common domain terms become ubiquitous across all documents, and length normalization, which penalizes central tables with more columns. By concatenating generated descriptions with original schema names, the retrieval system loses its ability to distinguish relevant objects. Sinha proposes a solution by separating fields into distinct indexes and using Reciprocal Rank Fusion (RRF) to combine results. This approach ensures that enrichment remains helpful without polluting the core search signals, providing a safer, more robust method for improving retrieval in text-to-SQL systems and similar document indexing tasks.