In time-distance helioseismology, information about the solar interior is encoded in measurements of travel times between pairs of points on the solar surface. Travel times are deduced from the cross-covariance of the random wave field. Here we consider travel times and also products of travel times as observables. They contain information about e.g. the statistical properties of convection in the Sun. The basic assumption of the model is that noise is the result of the stochastic excitation of solar waves, a random process which is stationary and Gaussian. We generalize the existing noise model (Gizon and Birch 2004) by dropping the assumption of horizontal spatial homogeneity. Using a recurrence relation, we calculate the noise covariance matrices for the moments of order 4, 6, and 8 of the observed wave field, for the moments of order 2, 3 and 4 of the cross-covariance, and for the moments of order 2, 3 and 4 of the travel times. All noise covariance matrices depend only on the expectation value of the cross-covariance of the observed wave field. For products of travel times, the noise covariance matrix consists of three terms proportional to $1/T$, $1/T^2$, and $1/T^3$, where $T$ is the duration of the observations. For typical observation times of a few hours, the term proportional to $1/T^2$ dominates and $Cov[tau_1 tau_2, tau_3 tau_4] approx Cov[tau_1, tau_3] Cov[tau_2, tau_4] + Cov[tau_1, tau_4] Cov[tau_2, tau_3]$, where the $tau_i$ are arbitrary travel times. This result is confirmed for $p_1$ travel times by Monte Carlo simulations and comparisons with SDO/HMI observations. General and accurate formulae have been derived to model the noise covariance matrix of helioseismic travel times and products of travel times. These results could easily be generalized to other methods of local helioseismology, such as helioseismic holography and ring diagram analysis.