We report an interference experiment of spontaneous emission of light from two distant solid-state ensembles of atoms that are coherently excited by a short laser pulse. The ensembles are Erbium ions doped into two LiNbO3 crystals with channel waveguides, which are placed in the two arms of a Mach-Zehnder interferometer. The light that is spontaneously emitted after the excitation pulse shows first-order interference. By a strong collective enhancement of the emission, the atoms behave as ideal two-level quantum systems and no which-path information is left in the atomic ensembles after emission of a photon. This results in a high fringe visibility of 95%, which implies that the observed spontaneous emission is highly coherent.