Efficient ab initio calculations of correlated materials at finite temperature require compact representations of the Greens functions both in imaginary time and Matsubara frequency. In this paper, we introduce a general procedure which generates sparse sampling points in time and frequency from compact orthogonal basis representations, such as Chebyshev polynomials and intermediate representation (IR) basis functions. These sampling points accurately resolve the information contained in the Greens function, and efficient transforms between different representations are formulated with minimal loss of information. As a demonstration, we apply the sparse sampling scheme to diagrammatic $GW$ and GF2 calculations of a hydrogen chain, of noble gas atoms and of a silicon crystal.