It is widely expected that at sufficiently high temperatures order is always lost, e.g. magnets loose their ferromagnetic properties. We pose the question of whether this is always the case in the context of quantum field theory in $d$ space dimensions. More concretely, one can ask whether there exist critical points (CFTs) which break some global symmetry at arbitrary finite temperature. The most familiar CFTs do not exhibit symmetry breaking at finite temperature, and moreover, in the context of the AdS/CFT correspondence, critical points at finite temperature are described by an uncharged black brane which obeys a no-hair theorem. Yet, we show that there exist CFTs which have some of their internal symmetries broken at arbitrary finite temperature. Our main example is a vector model which we study both in the epsilon expansion and arbitrary rank as well as the large rank limit (and arbitrary dimension). The large rank limit of the vector model displays a conformal manifold, a moduli space of vacua, and a deformed moduli space of vacua at finite temperature. The appropriate Nambu-Goldstone bosons including the dilaton-like particle are identified. Using these tools we establish symmetry breaking at finite temperature for finite small $epsilon$. We also prove that a large class of other fixed points, which describe some of the most common quantum magnets, indeed behave as expected and do not break any global symmetry at finite temperature. We discuss some of the consequences of finite temperature symmetry breaking for the spectrum of local operators. Finally, we propose a class of fixed points which appear to be possible candidates for finite temperature symmetry breaking in $d=2$.