nginx streams your tokens fine. HAProxy holds them for 206ms.

A recent performance analysis investigates the common assumption that nginx buffers Server-Sent Events (SSE) streams, often leading developers to disable proxy_buffering. The study, which utilized a controlled benchmarking harness, reveals that nginx does not inherently buffer token streams. In contrast, HAProxy in its default configuration introduces a significant 206ms latency for the first token, coalescing small frames into larger bursts. The author demonstrates that the widely recommended 'X-Accel-Buffering: no' header is an nginx-specific convention that has no effect on HAProxy. While proxy_buffering can cause minor latency in specific bulk-payload scenarios, it is not the primary cause of token-streaming delays. The findings emphasize that for LLM token streaming, HAProxy's default behavior is the actual bottleneck, rather than the commonly blamed nginx settings. The author provides a detailed methodology to ensure measurement accuracy, highlighting the importance of testing with realistic payload sizes.