The Price of Sovereignty: Running AliceAI-Foundation-80B

The author shares their personal experience attempting to run the new AliceAI-Foundation-80B language model from Yandex. The focus is on the practical difficulties of renting the computing power required for inference of a model of this scale. The author opted against quantization and CPU-offload to ensure maximum performance, which necessitated the use of four NVIDIA A100 GPUs. During testing, significant issues arose regarding quota availability in cloud services, including Datasphere. Despite attempts to configure virtual machines and utilize various cloud infrastructure tools, the process proved technically challenging and costly. The article raises an important question about infrastructure accessibility for independent researchers and developers wanting to work with large local language models, highlighting the gap between the advertised capabilities of cloud platforms and the actual user experience when attempting to deploy heavy AI solutions.