Let M be the moduli scheme of canonically polarized manifolds with Hilbert polynomial h. We construct for a given finite set I of natural numbers m>1 with h(m)>0 a projective compactification M of the reduced scheme underlying M such that the ample invertible sheaf L corresponding to the determinant of the direct image of the m-th power of the relative dualizing sheaf on the moduli stack, has a natural extension L to M. A similar result is shown for moduli of polarized minimal models of Kodaira dimension zero. In both cases natural means that the pullback of L to a curve C --> M, induced by a family f:X --> C is isomorphic to the determinant of the direct image of the m-th power of the relative dualizing sheaf whenever f is birational to a semi-stable family. Besides of the weak semistable reduction of Abramovich-Karu and the extension theorem of Gabber there are new tools, hopefully of interest by itself. In particular we will need a theorem on the flattening of multiplier sheaves in families, on their compatibility with pullbacks and on base change for their direct images, twisted by certain semiample sheaves. Following suggestions of a referee, we reorganized the article, we added several comments explaining the main line of the proof, and we changed notations a little bit.