In this paper, we draw attention to a problem that is often overlooked or ignored by companies practicing hypothesis testing (A/B testing) in online environments. We show that conducting experiments on limited inventory that is shared between variant
s in the experiment can lead to high false positive rates since the core assumption of independence between the groups is violated. We provide a detailed analysis of the problem in a simplified setting whose parameters are informed by realistic scenarios. The setting we consider is a $2$-dimensional random walk in a semi-infinite strip. It is rich enough to take a finite inventory into account, but is at the same time simple enough to allow for a closed form of the false-positive probability. We prove that high false-positive rates can occur, and develop tools that are suitable to help design adequate tests in follow-up work. Our results also show that high false-negative rates may occur. The proofs rely on a functional limit theorem for the $2$-dimensional random walk in a semi-infinite strip.
We study the asymptotic behavior as $t to infty$ of a time-dependent family $(mu_t)_{t geq 0}$ of probability measures on $mathbb{R}$ solving the kinetic-type evolution equation $partial_t mu_t + mu_t = Q(mu_t)$ where $Q$ is a smoothing transformatio
n on $mathbb{R}$. This problem has been investigated earlier, e.g. by Bassetti and Ladelli [Ann. Appl. Probab. 22(5): 1928-1961, 2012] and Bogus, Buraczewski and Marynych [Stochastic Process. Appl. 130(2):677-693, 2020]. Combining the refined analysis of the latter paper, which provides a probabilistic description of the solution $mu_t$ as the law of a suitable random sum related to a continuous-time branching random walk at time $t$, with recent advances in the analysis of the extremal positions in the branching random walk we are able to solve the remaining case that has been left open until now. In the course of our work, we significantly weaken the assumptions in the literature that guarantee the existence (and uniqueness) of a solution to the evolution equation $partial_t mu_t + mu_t = Q(mu_t)$.
