Dust-enshrouded, starbursting, submillimeter galaxies (SMGs) at $z geq 3$ have been proposed as progenitors of $z geq 2$ compact quiescent galaxies (cQGs). To test this connection, we present a detailed spatially resolved study of the stars, dust and stellar mass in a sample of six submillimeter-bright starburst galaxies at $z sim 4.5$. The stellar UV emission probed by HST is extended, irregular and shows evidence of multiple components. Informed by HST, we deblend Spitzer/IRAC data at rest-frame optical finding that the systems are undergoing minor mergers, with a typical stellar mass ratio of 1:6.5. The FIR dust continuum emission traced by ALMA locates the bulk of star formation in extremely compact regions (median $r_{rm{e}} = 0.70 pm 0.29$ kpc) and it is in all cases associated with the most massive component of the mergers (median $log (M_{*}/M_{odot}) = 10.49 pm 0.32$). We compare spatially resolved UV slope ($beta$) maps with the FIR dust continuum to study the infrared excess ($rm{IRX} = L_{rm{IR}}/L_{rm{UV}}$)-$beta$ relation. The SMGs display systematically higher $rm{IRX}$ values than expected from the nominal trend, demonstrating that the FIR and UV emissions are spatially disconnected. Finally, we show that the SMGs fall on the mass-size plane at smaller stellar masses and sizes than cQGs at $z = 2$. Taking into account the expected evolution in stellar mass and size between $z = 4.5$ and $z = 2$ due to the ongoing starburst and mergers with minor companions, this is in agreement with a direct evolutionary connection between the two populations.