The formation of temporal dissipative solitons in optical microresonators enables compact, high repetition rate sources of ultra-short pulses as well as low noise, broadband optical frequency combs with smooth spectral envelopes. Here we study the influence of the resonator mode spectrum on temporal soliton formation. Using frequency comb assisted diode laser spectroscopy, the measured mode structure of crystalline MgF2 resonators are correlated with temporal soliton formation. While an overal general anomalous dispersion is required, it is found that higher order dispersion can be tolerated as long as it does not dominate the resonators mode structure. Mode coupling induced avoided crossings in the resonator mode spectrum are found to prevent soliton formation, when affecting resonator modes close to the pump laser. The experimental observations are in excellent agreement with numerical simulations based on the nonlinear coupled mode equations, which reveal the rich interplay of mode crossings and soliton formation.