"Let's just deploy our own 1T parameter LLM": what had to be built around it at Bitrix24

This article examines the technical and architectural challenges faced by the Bitrix24 team when integrating a large language model (LLM) with 1 trillion parameters into their ecosystem. The author emphasizes that the phrase "just deploy" hides complex engineering work related to infrastructure management. The focus is on allocating computing power among various services, such as chats, long document processing, and AI agents. The article also discusses resource optimization methods during off-peak periods, request prioritization strategies, and testing systems designed to ensure that model updates do not degrade response quality. Based on a presentation from the Saint HighLoad++ conference, the article provides a systematic overview of building a scalable platform for generative AI within a large enterprise product.