We present results for the phase diagram of an SU($N$) generalization of the Heisenberg antiferromagnet on a bipartite three-dimensional anisotropic cubic (tetragonal) lattice as a function of $N$ and the lattice anisotropy $gamma$. In the isotropic $gamma=1$ cubic limit, we find a transition from N{e}el to valence bond solid (VBS) between N=9 and N=10. We follow the N{e}el-VBS transition to the limiting cases of $gamma ll 1 $ (weakly coupled layers) and $gamma gg 1$ (weakly coupled chains). Throughout the phase diagram we find a direct first-order transition from N{e}el at small-$N$ to VBS at large-$N$. In the three-dimensional models studied here, we find no evidence for either an intervening spin-liquid photon phase or a continuous transition, even close to the limit $gamma ll 1$ where the isolated layers undergo continuous N{e}el-VBS transitions.