We present an extensive quantum Monte Carlo study of the Neel-valence bond solid (VBS) phase transition on rectangular and honeycomb lattice SU($N$) antiferromagnets in sign problem free models. We find that in contrast to the honeycomb lattice and previously studied square lattice systems, on the rectangular lattice for small $N$ a first order Neel-VBS transition is realized. On increasing $Ngeq 4$, we observe that the transition becomes continuous and with the {em same} universal exponents as found on the honeycomb and square lattices (studied here for $N=5,7,10$), providing strong support for a deconfined quantum critical point. Combining our new results with previous numerical and analytical studies we present a general phase diagram of the stability of $mathbb{CP}^{N-1}$ fixed points with $q$-monopoles.