How we taught LLMCOD to build a knowledge base from SPAs and open APIs, not just HTML

The developers of the LLMCOD service have introduced an update that expands the capabilities of building knowledge bases for RAG systems. The traditional method of parsing HTML code is becoming ineffective for modern SPA sites, where content is dynamically loaded via JavaScript and APIs. The LLMCOD team has implemented a mechanism that allows the system to interact with APIs and process data that does not appear in the page's source code. The article details the technical challenges the developers faced while integrating these methods and explains why vector search alone was insufficient for high-quality information extraction. The new solution allows for more efficient indexing of complex web resources, ensuring data completeness for LLM operations. The authors share their experience in overcoming the limitations of classic scraping and emphasize the importance of adapting tools to modern web application architectures.