We employ polarization-resolved Raman spectroscopy to study multi-band stoichiometric superconductor CaKFe$_4$As$_4$. The B$_{2g}$ symmetry Raman response shows no signatures of Pomeranchuk-like electronic nematic fluctuations which is observed for many other Fe-based superconductors. In the superconducting state, we identify three pair-breaking peaks at 13.8, 16.9 and 21 meV and full spectral weight suppression at low energies. The pair-breaking peak energies in Raman response are about 20% lower than twice the gap energies as measured by single-particle spectroscopy, implying a sub-dominant $d$-wave symmetry interaction. We analyze the superconductivity induced phonon self-energy effects and give an estimation of weak electron-phonon coupling constant $lambda^Gamma$=0.0015.