We present a systematic optical study for a bandwidth-controlled series of nearly half doped colossal magnetoresistive manganites RE$_{0.55}$AE$_{0.45}$MnO$_3$ (RE and AE being rare earth and alkaline earth ions, respectively) under the presence of quenched disorder over a broad temperature region $T=10-800$ K. The ground state of the compounds ranges from the charge and orbital ordered insulator through the spin glass to the ferromagnetic metal. The enhanced phase fluctuations, namely the short-range charge and orbital correlations dominate the paramagnetic region of the phase diagram above all the ground-state phases. This paramagnetic region is characterized by a full-gap to pseudo-gap crossover towards elevated temperatures where a broad low-energy electronic structure appears in the conductivity spectra over a large variation of the bandwidth. This pseudo-gap state with local correlations is robust against thermal fluctuations at least up to T=800 K. For small bandwidth the onset of the long-range charge order is accompanied by an instantaneous increase of the gap. The emergence of the ferromagnetic state is manifested in the optical spectra as a first-order insulator to metal transition for compounds with moderate bandwidth while it becomes a second-order transition on the larger bandwidth side. Unusually large scattering rate of the metallic carriers is observed in the ferromagnetic state which is attributed to orbital correlation with probably rod-like ($3z^2-r^2$-like) character.