We study the three-dimensional $U(N)$ Gross-Neveu and CP$^{N-1}$ models in the canonical formalism with fixed $U(1)$ charge. For large-$N$ this is closely related to coupling the models to abelian Chern-Simons in a monopole background. We show that the presence of the imaginary chemical potential for the $U(1)$ charge makes the phase structure of the models remarkably similar. We calculate their respective large-$N$ free energy densities and show that they are mapped into each other in a precise way. Intriguingly, the free energy map involves the Bloch-Wigner function and its generalizations introduced by Zagier. We expect that our results are connected to the recently discussed $3d$ bosonization.