Low-defect two-dimensional electron systems (2DESs) are essential for studies of fragile many-body interactions that only emerge in nearly-ideal systems. As a result, numerous efforts have been made to improve the quality of modulation-doped Al$_x$Ga$_{1-x}$As/GaAs quantum wells (QWs), with an emphasis on purifying the source material of the QW itself or achieving better vacuum in the deposition chamber. However, this approach overlooks another crucial component that comprises such QWs, the Al$_x$Ga$_{1-x}$As barrier. Here we show that having a clean Al source and hence a clean barrier is instrumental to obtain a high-quality GaAs 2DES in a QW. We observe that the mobility of the 2DES in GaAs QWs declines as the thickness or Al content of the Al$_x$Ga$_{1-x}$As barrier beneath the QW is increased, which we attribute to the surface segregation of Oxygen atoms that originate from the Al source. This conjecture is supported by the improved mobility in the GaAs QWs as the Al cell is cleaned out by baking.