The magnetic and electrical properties of high quality single crystals of $A$-site disordered (solid solution) Ln$_{0.5}$Ba$_{0.5}$MnO$_3$ are investigated near the phase boundary between the spin glass insulator and colossal-magnetoresistive ferromagnetic metal, locating near Ln = Sm. The temperature dependence of the ac-susceptibility and the x-ray diffuse scattering of Eu$_{0.5}$Ba$_{0.5}$MnO$_3$ are analyzed in detail. The uniformity of the random potential perturbation in Ln$_{0.5}$Ba$_{0.5}$MnO$_3$ crystals with small bandwidth yields, rather than the phase separation, an homogeneous short ranged charge/orbital order which gives rise to a nearly-atomic spin glass state. Remarkably, this microscopically disordered ``CE-glass state alone is able to bring forth the colossal magnetoresistance.