Time-resolved transmittance measurements performed on Ga$_{0.94}$Mn$_{0.06}$As in the vicinity of the Mn-induced mid-infrared absorption band are presented. Upon photo-excitation, a slow increase (hundreds of ps timescale) of the differential transmittance is observed and found to be directly related to demagnetization. The temporal profiles of the transmittance and of the demagnetization measured by time-resolved magneto-optical Kerr spectroscopy are found to coincide. Well below the Curie temperature, the maximum amplitude of the slow component of the differential transmittance as a function of the probe energy is on the rising edge of the linear absorption peak, suggesting that ferromagnetic ordering can be explained by a coupling of the Mn local spins through bound magnetic polarons.