Sizing RAM and vCPU for a Local Language Model: Calculating the Starting VM Size

This article explores the methodology for selecting hardware resources to deploy local language models (LLMs). The author emphasizes that the choice of RAM and vCPU configuration depends directly on model weights, request profiles, context length, and the offload strategy employed. The material provides an approach for calculating baseline requirements based on model weights and the KV-cache formula. Special attention is given to the importance of load testing after initial setup: a 'warmed-up' test reveals actual memory consumption and identifies the performance threshold of the CPU. This guide helps engineers avoid over-provisioning resources and optimize costs when launching self-hosted solutions. The article serves as a useful manual for professionals involved in implementing generative AI within corporate infrastructure, where stability and predictability of model performance are critical.