We present a combined study from angle-resolved photoemission and density-functional theory calculations of the temperature-dependent electronic structure in the excitonic insulator candidate Ta$_2$NiSe$_5$. Our experimental measurements unambiguously establish the normal state as a semimetal with a significant band overlap of $>$100~meV. Our temperature-dependent measurements indicate how these low-energy states hybridise when cooling through the well-known 327~K phase transition in this system. From our calculations and polarisation-dependent photoemission measurements, we demonstrate the importance of a loss of mirror symmetry in enabling the band hybridisation, driven by a shear-like structural distortion which reduces the crystal symmetry from orthorhombic to monoclinic. Our results thus point to the key role of the lattice distortion in enabling the phase transition of Ta$_2$NiSe$_5$.