We study theoretically the role of carrier multiplication due to impact ionization after an ultrafast optical excitation in a model system of a quasi-two dimensional material with a small band gap. As a mechanism for the photo-induced band gap narrowing we use coherent phonons, which mimics the quenching of an insulator phase. We discuss the importance of impact ionization in the ultrafast response, and investigate the interplay between carrier and band dynamics. Our model allows us to compare with recent experiments and identify signatures of carrier multiplication in typical electronic distribution curves as measured by time-resolved photoemission spectroscopy. In particular we investigate the influence of the shape of the bands on the carrier multiplication and the respective contributions of band and carrier dynamics to electronic distribution curves.