The envisioned existence of an excitonic-insulator phase in Ta$_2$NiSe$_5$ has attracted a remarkable interest in this material. The origin of the phase transition in Ta$_2$NiSe$_5$ has been rationalized in terms of crystal symmetries breaking driven by both electronic correlation and lattice distortion. However, the role of structural and electronic effects has yet to be disentangled. Meanwhile its complementary material Ta$_2$NiS$_5$, which has the chalcogen species exchanged with Sulfur, does not show any experimental evidence of an excitonic insulating phase. Here we present a microscopic investigation of the electronic and phononic effects involved in the structural phase transition in Ta$_2$NiSe$_5$ and Ta$_2$NiS$_5$ by means of extensive first-principles calculations for both the high temperature orthorhombic and low-temperature monoclinic crystal phases. We show that, despite the difference in electronic behaviour, the structural origin of the phase transition is the same in the two crystals. In particular our first-principles results suggest, that the high temperature phase of Ta$_2$NiSe$_5$ is metallic and the structural transition to the low-temperature phase leads to the opening of an electronic gap. By analysing the phononic modes of the two phases we single out the mode responsible for the structural transition and demonstrate how this phonon mode strongly couples to the electronic structure. We demonstrate that, despite the very similar phononic behaviour, in Ta$_2$NiS$_5$ the electronic transition from metal to semiconductor is lacking and the crystal remains a semiconductor in both phases. To disentangle the effect of electronic correlation, we calculate electronic bandstructures with increasing accuracy in the electron-electron interaction and find that the structural transition alone allows for the metal to semiconductor phase transition, ...