Iron-based compounds (IBS) display a surprising variety of superconducting properties that seems to arise from the strong sensitivity of these systems to tiny details of the lattice structure. In this respect, systems that become superconducting under pressure, like CaFe$_2$As$_2$, are of particular interest. Here we report on the first directional point-contact Andreev-reflection spectroscopy (PCARS) measurements on CaFe$_2$As$_2$ crystals under quasi-hydrostatic pressure, and on the interpretation of the results using a 3D model for Andreev reflection combined with ab-initio calculations of the Fermi surface (within the density functional theory) and of the order parameter symmetry (within a random-phase-approximation approach in a ten-orbital model). The almost perfect agreement between PCARS results at different pressures and theoretical predictions highlights the intimate connection between the changes in the lattice structure, a topological transition in the hole-like Fermi surface sheet, and the emergence on the same sheet of an order parameter with a horizontal node line.