Pr$_2$Zr$_2$O$_7$ is a pyrochlore quantum spin-ice candidate. Using Raman scattering spectroscopy we probe crystal electric field excitations of Pr$^{3+}$, and demonstrate the importance of their interactions with the lattice. We identify a vibronic interaction with a phonon that leads to a splitting of a doublet crystal field excitation at around 55~meV. We also probe a splitting of the non-Kramers ground state doublet of Pr$^{3+}$ by observing a double line of the excitations to the first excited singlet state $E^0_g rightarrow A_{1g}$. We show that the splitting has a strong temperature dependence, with the doublet structure most prominent between 50~K and 100~K, and the weight of one of the components strongly decreases on cooling. We suggest a static or dynamic deviation of Pr$^{3+}$ from the position in the ideal crystal structure can be the origin of the effect, with the deviation strongly decreasing at low temperatures.