The recent experimental condensation of ultracold atoms in a triangular optical lattice with negative effective tunneling energies paves the way to study frustrated systems in a controlled environment. Here, we explore the critical behavior of the chiral phase transition in such a frustrated lattice in three dimensions. We represent the low-energy action of the lattice system as a two-component Bose gas corresponding to the two minima of the dispersion. The contact repulsion between the bosons separates into intra- and inter-component interactions, referred to as $V_{0}$ and $V_{12}$, respectively. We first employ a Huang-Yang-Luttinger approximation of the free energy. For $V_{12}/V_{0} = 2$, which corresponds to the bare interaction, this approach suggests a first order phase transition, at which both the U$(1)$ symmetry of condensation and the $mathbb{Z}_2$ symmetry of the emergent chiral order are broken simultaneously. Furthermore, we perform a renormalization group calculation at one-loop order. We demonstrate that the coupling regime $0<V_{12}/V_0leq1$ shares the critical behavior of the Heisenberg fixed point at $V_{12}/V_{0}=1$. For $V_{12}/V_0>1$ we show that $V_{0}$ flows to a negative value, while $V_{12}$ increases and remains positive. This results in a breakdown of the effective quartic field theory due to a cubic anisotropy, and again suggests a discontinuous phase transition.
Download