We present new magnetic heat capacity and neutron scattering results for two magnetically frustrated molybdate pyrochlores: $S=1$ oxide Lu$_2$Mo$_2$O$_7$ and $S={frac{1}{2}}$ oxynitride Lu$_2$Mo$_2$O$_5$N$_2$. Lu$_2$Mo$_2$O$_7$ undergoes a transition to an unconventional spin glass ground state at $T_f {sim} 16$ K. However, the preparation of the corresponding oxynitride tunes the nature of the ground state from spin glass to quantum spin liquid. The comparison of the static and dynamic spin correlations within the oxide and oxynitride phases presented here reveals the crucial role played by quantum fluctuations in the selection of a ground state. Furthermore, we estimate an upper limit for a gap in the spin excitation spectrum of the quantum spin liquid state of the oxynitride of ${Delta} {sim} 0.05$ meV or ${frac{Delta}{|theta|}}sim0.004$, in units of its antiferromagnetic Weiss constant ${theta} {sim}-121$ K.