The chiral langasite Ba$_3$NbFe$_3$Si$_2$O$_{14}$ is a multiferroic compound. While its magnetic order below T$_N$=27 K is now well characterised, its polar order is still controversial. We thus looked at the phonon spectrum and its temperature dependence to unravel possible crystal symmetry breaking. We combined optical measurements (both infrared and Raman spectroscopy) with ab initio calculations and show that signatures of a polar state are clearly present in the phonon spectrum even at room temperature. An additional symmetry lowering occurs below 120~K as seen from emergence of softer phonon modes in the THz range. These results confirm the multiferroic nature of this langasite and open new routes to understand the origin of the polar state.