Recently, much attention has been focused on the replicability of scientific results, causing scientists, statisticians, and journal editors to examine closely their methodologies and publishing criteria. Experimental particle physicists have been aw
are of the precursors of non-replicable research for many decades and have many safeguards to ensure that the published results are as reliable as possible. The experiments require large investments of time and effort to design, construct, and operate. Large collaborations produce and check the results, and many papers are signed by more than three thousand authors. This paper gives an introduction to what experimental particle physics is and to some of the tools that are used to analyze the data. It describes the procedures used to ensure that results can be computationally reproduced, both by collaborators and by non-collaborators. It describes the status of publicly available data sets and analysis tools that aid in reproduction and recasting of experimental results. It also describes methods particle physicists use to maximize the reliability of the results, which increases the probability that they can be replicated by other collaborations or even the same collaborations with more data and new personnel. Examples of results that were later found to be false are given, both with failed replication attempts and one with alarmingly successful replications. While some of the characteristics of particle physics experiments are unique, many of the procedures and techniques can be and are used in other fields.
A short overview is provided of the recent PHYSTAT$ u$ meeting at CERN, which dealt with statistical issues relevant for neutrino experiments.
We consider whether the asymptotic distributions for the log-likelihood ratio test statistic are expected to be Gaussian or chi-squared. Two straightforward examples provide insight on the difference.
This is a summary of the recent PHYSTAT-$ u$ Workshops in Japan and at Fermilab, on `Statistical Issues in Experimental Neutrino Physics.
We review the methods to combine several measurements, in the form of parameter values or $p$-values.
Various statistical issues relevant to searches for new physics or to parameter determination in analyses of data in neutrino experiments are briefly discussed.
Given the cost, both financial and even more importantly in terms of human effort, in building High Energy Physics accelerators and detectors and running them, it is important to use good statistical techniques in analysing data. Some of the statisti
cal issues that arise in searches for New Physics are discussed briefly. They include topics such as: Should we insist on the 5 sigma criterion for discovery claims? The probability of A, given B, is not the same as the probability of B, given A. The meaning of p-values. What is Wilks Theorem and when does it not apply? How should we deal with the `Look Elsewhere Effect? Dealing with systematics such as background parametrisation. Coverage: What is it and does my method have the correct coverage? The use of p0 versus p1 plots.
For situations where we are trying to decide which of two hypotheses $H_{0}$ and $H_{1}$ provides a better description of some data, we discuss the usefulness of plots of $p_{0}$ versus $p_{1}$, where $p_{i}$ is the $p$-value for testing $H_{i}$. The
y provide an interesting way of understanding the difference between the standard way of excluding $H_{1}$ and the $CL_{s}$ approach; the Punzi definition of sensitivity; the relationship between $p$-values and likelihood ratios; and the probability of observing misleading evidence. They also help illustrate the Law of the Iterated Logarithm and the Jeffreys-Lindley paradox.
We consider the relative merits of two different approaches to discovery or exclusion of new phenomena, a raster scan or a 2-dimensional approach.
We discuss the traditional criterion for discovery in Particle Physics of requiring a significance corresponding to at least 5 sigma; and whether a more nuanced approach might be better.
