Your LLM Telemetry Table Does Not Have One Denominator

In a recent technical note, the author cautions against the common pitfall of treating LLM telemetry data as monolithic. When analyzing performance metrics across different models, developers often mistakenly aggregate disparate units of measurement—such as thread-level completion proxies and epoch-attributed turn fragments—into a single table. This practice creates misleading comparisons because the denominators for these metrics are not interchangeable. The author argues that a model name is not a stable unit of analysis and that telemetry reports must account for variables like role, task family, and dispatch policy. By failing to distinguish between these strata, engineers risk generating 'story-driven' data rather than actionable insights. The article concludes by proposing a rigorous framework for recording telemetry, emphasizing that observed performance gaps should be treated as associations rather than causal evidence until controlled experiments are conducted to validate the findings.