Speckle interferometry is an established optical metrology tool for the characterization of rough objects. The raw phase, however, is impaired by the presence of phase singularities, making the unwrapping procedure ambiguous. In a Michelson setup, we tailor the spatial coherence of the light source, achieving a physical averaging of independent, mutually incoherent speckle fields. In the resulting raw phase, the systematic phase is preserved while the number of phase singularities is greatly reduced. Both interferometer arms are affected by the averaging. The reduction is sufficient to even allow the use of a standard unwrapping algorithm originally developed for smooth surfaces only.