Scoring Functions for Multivariate Distributions and Level Sets

Interest in predicting multivariate probability distributions is growing due to the increasing availability of rich datasets and computational developments. Scoring functions enable the comparison of forecast accuracy, and can potentially be used for estimation. A scoring function for multivariate distributions that has gained some popularity is the energy score. This is a generalization of the continuous ranked probability score (CRPS), which is widely used for univariate distributions. A little-known, alternative generalization is the multivariate CRPS (MCRPS). We propose a theoretical framework for scoring functions for multivariate distributions, which encompasses the energy score and MCRPS, as well as the quadratic score, which has also received little attention. We demonstrate how this framework can be used to generate new scores. For univariate distributions, it is well-established that the CRPS can be expressed as the integral over a quantile score. We show that, in a similar way, scoring functions for multivariate distributions can be disintegrated to obtain scoring functions for level sets. Using this, we present scoring functions for different types of level set, including those for densities and cumulative distributions. To compute the scoring functions, we propose a simple numerical algorithm. We illustrate our proposals using simulated and stock returns data.