Pb$_{0.77}$Sn$_{0.23}$Se is a novel alloy of two promising thermoelectric materials PbSe and SnSe that exhibits a temperature dependent band inversion below 300 K. Recent work has shown that this band inversion also coincides with a trivial to nontrivial topological phase transition. To understand how the properties critical to thermoelectric efficiency are affected by the band inversion, we measured the broadband optical response of Pb$_{0.77}$Sn$_{0.23}$Se as a function of temperature. We find clear optical evidence of the band inversion at $160pm15$ K, and use the extended Drude model to accurately determine a $T^{3/2}$ dependence of the bulk carrier lifetime, associated with electron-acoustic phonon scattering. Due to the high bulk carrier doping level, no discriminating signatures of the topological surface states are found, although their presence cannot be excluded from our data.