Using time-resolved optical Kerr rotation, we measure the low temperature valley dynamics of resident electrons and holes in exfoliated WSe$_2$ monolayers as a systematic function of carrier density. In an effort to reconcile the many disparate timescales of carrier valley dynamics in monolayer semiconductors reported to date, we directly compare the doping-dependent valley relaxation in two electrostatically-gated WSe$_2$ monolayers having different dielectric environments. In a fully-encapsulated structure (hBN/WSe$_2$/hBN, where hBN is hexagonal boron nitride), valley relaxation is found to be monoexponential. The valley relaxation time $tau_v$ is quite long ($sim$10~$mu$s) at low carrier densities, but decreases rapidly to less than 100~ns at high electron or hole densities $gtrsim$2 $times 10^{12}$~cm$^{-2}$. In contrast, in a partially-encapsulated WSe$_2$ monolayer placed directly on silicon dioxide (hBN/WSe$_2$/SiO$_2$), carrier valley relaxation is multi-exponential at low carrier densities. The difference is attributed to environmental disorder from the SiO$_2$ substrate. Unexpectedly, very small out-of-plane magnetic fields can increase $tau_v$, especially in the hBN/WSe$_2$/SiO$_2$ structure, suggesting that localized states induced by disorder can play an important role in depolarizing spins and mediating the valley relaxation of resident carriers in monolayer transition metal-dichalcogenide semiconductors.