Within a galaxy the stellar mass-to-light ratio $Upsilon_*$ is not constant. Spatially resolved kinematics of nearby early-type galaxies suggest that allowing for a variable initial mass function (IMF) returns significantly larger $Upsilon_*$ gradients than if the IMF is held fixed. If $Upsilon_*$ is greater in the central regions, then ignoring the IMF-driven gradient can overestimate $M_*^{rm dyn}$ by as much as a factor of two for the most massive galaxies, though stellar population estimates $M_*^{rm SP}$ are also affected. Large $Upsilon_*$-gradients have four main consequences: First, $M_*^{rm dyn}$ cannot be estimated independently of stellar population synthesis models. Second, if there is a lower limit to $Upsilon_*$ and gradients are unknown, then requiring $M_*^{rm dyn}=M_*^{rm SP}$ constrains them. Third, if gradients are stronger in more massive galaxies, then $M_*^{rm dyn}$ and $M_*^{rm SP}$ can be brought into agreement, not by shifting $M_*^{rm SP}$ upwards by invoking constant bottom-heavy IMFs, as advocated by a number of recent studies, but by revising $M_*^{rm dyn}$ estimates in the literature downwards. Fourth, accounting for $Upsilon_*$ gradients changes the high-mass slope of the stellar mass function $phi(M_*^{rm dyn})$, and reduces the associated stellar mass density. These conclusions potentially impact estimates of the need for feedback and adiabatic contraction, so our results highlight the importance of measuring $Upsilon_*$ gradients in larger samples.