Which Type of Statistical Uncertainty Helps Evidence-Based Policymaking? An Insight from a Survey Experiment in Ireland

Which type of statistical uncertainty -- Frequentist statistical (in)significance with a p-value, or a Bayesian probability -- helps evidence-based policymaking better? To investigate this, I ran a survey experiment on a sample from the population of Ireland and obtained 517 responses. The experiment asked these participants to decide to or not to introduce a new bus line as a policy to reduce traffic jams. The treatment was the different types of statistical uncertainty information: statistical (in)significance with a p-value, and the probability that the estimate is correct. In each type, uncertainty was set either low or non-low. It turned out that participants shown the Frequentist information exhibited a much more deterministic tendency to adopting or not adopting the policy than those shown the Bayesian information, given the actual difference between the low-uncertainty and non-low-uncertainty the experimental scenarios implied. This finding suggests that policy-relevant quantitative research should present the uncertainty of statistical estimates using the probability of associated policy effects rather than statistical (in)significance, to allow the general public and policymakers to correctly evaluate the continuous nature of statistical uncertainty.