Close-packed, classical dimer models on three-dimensional, bipartite lattices harbor a Coulomb phase with power-law correlations at infinite temperature. Here, we discuss the nature of the thermal phase transition out of this Coulomb phase for a variety of dimer models which energetically favor crystalline dimer states with columnar ordering. For a family of these models we find a direct thermal transition from the Coulomb phase to the dimer crystal. While some systems exhibit (strong) first-order transitions in correspondence with the Landau-Ginzburg-Wilson paradigm, we also find clear numerical evidence for continuous transitions. A second family of models undergoes two consecutive thermal transitions with an intermediate paramagnetic phase separating the Coulomb phase from the dimer crystal. We can describe all of these phase transitions in one unifying framework of candidate field theories with two complex Ginzburg-Landau fields coupled to a U(1) gauge field. We derive the symmetry-mandated Ginzburg-Landau actions in these field variables for the various dimer models and discuss implications for their respective phase transitions.