The emission of prompt fission $gamma$ rays within a few nanoseconds to a few microseconds following the scission point is studied in the Hauser-Feshbach formalism applied to the deexcitation of primary excited fission fragments. Neutron and $gamma$-ray evaporations from fully accelerated fission fragments are calculated in competition at each stage of the decay, and the role of isomers in the fission products, before $beta$-decay, is analyzed. The time evolution of the average total $gamma$-ray energy, average total $gamma$-ray multiplicity, and fragment-specific $gamma$-ray spectra, is presented in the case of neutron-induced fission reactions of $^{235}$U and $^{239}$Pu, as well as spontaneous fission of $^{252}$Cf. The production of specific isomeric states is calculated and compared to available experimental data. About 7% of all prompt fission $gamma$ rays are predicted to be emitted between 10 nsec and 5 $mu$sec following fission, in the case of $^{235}$U and $^{239}$Pu $(n_{rm th},f)$ reactions, and up to 3% in the case of $^{252}$Cf spontaneous fission. The cumulative average total $gamma$-ray energy increases by 2 to 5% in the same time interval. Finally, those results are shown to be robust against significant changes in the model input parameters.