Many data-analysis problems involve large dense matrices that describe the covariance of stationary noise processes; the computational cost of inverting these matrices, or equivalently of solving linear systems that contain them, is often a practical limit for the analysis. We describe two general, practical, and accurate methods to approximate stationary covariance matrices as low-rank matrix products featuring carefully chosen spectral components. These methods can be used to greatly accelerate data-analysis methods in many contexts, such as the Bayesian and generalized-least-squares analysis of pulsar-timing residuals.