A Guide to Switchback Experiments: Window Length, Error Clustering, and Power

This Habr article explores the methodology of switchback experiments, which are essential for evaluating changes in systems with shared resources where standard user-based A/B testing can lead to biased results. The author explains why traditional approaches often yield false conclusions and provides methods for accurate performance assessment. The material covers key experimental parameters: selecting the optimal switching window length, techniques for error clustering, and calculating statistical power. Understanding these aspects helps avoid pitfalls when deploying new algorithms and ensures that the expected impact aligns with real business metrics. This article is a valuable resource for data analysts and product developers who face the limitations of classical testing methods in environments with high resource contention.