We present observations of background charge fluctuators near an Al-AlO_x-Al single-electron transistor on an oxidized Si substrate. The transistor design incorporates a heavily doped substrate and top gate, which allow for independent control of the substrate and transistor island potentials. Through controlled charging of the Si/SiO_2 interface we show that the fluctuators cannot reside in the Si layer or in the tunnel barriers. Combined with the large measured signal amplitude, this implies that the defects must be located very near the oxide surface.