What is disaggregated prefill and decode in LLM inference?

LLM inference consists of two distinct phases: prefill, which is compute-bound and determines Time to First Token (TTFT), and decode, which is memory-bandwidth-bound and dictates Inter-Token Latency (ITL). Traditionally, these phases are colocated on the same GPUs, leading to resource contention, latency spikes, and jitter. The article explores the concept of disaggregated serving, where prefill and decode tasks are offloaded to separate GPU pools. By decoupling these processes, developers can optimize hardware utilization for each phase independently, significantly improving performance for high-concurrency applications. While this architecture introduces the need for efficient KV cache transfers between pools, it has become a standard pattern for production-grade LLM serving. The piece highlights how managed services like DigitalOcean Serverless Inference leverage this approach to maintain strict streaming Service Level Objectives (SLOs) and provides guidance for those looking to implement similar architectures on self-hosted GPU infrastructure.