Identical data, different p-values: why the stopping rule matters

In this Habr article, the author explores a fundamental issue in statistical inference: how the experiment stopping rule affects p-values. Using a coin-tossing example, the author demonstrates why identical datasets can yield different statistical results depending on when the researcher decides to stop collecting data. The focus is on the problem of 'peeking' in A/B testing. The author explains that the error lies not in viewing intermediate results, but in how those results influence the decision to terminate the experiment. This practice increases the likelihood of false positives and distorts conclusions. The article is valuable for analysts and professionals conducting experiments, as it clarifies the mathematical nature of errors in interpreting statistical significance and emphasizes the importance of strictly adhering to experimental design methodologies before testing begins.