We report Raman-scattering results of YbRu$_2$Ge$_2$ single crystals to explore the phononic and crystal-field (CF) excitations. This heavy-fermion metal is suggested to enter a ferroquadrupolar (FQ) phase below T$_0$=10 K. The tetragonal CF potential splits the Yb$^{3+}$ $^2F_{7/2}$ ground multiplet into two $Gamma_6$ and two $Gamma_7$ Kramers doublets. We establish the following CF level scheme of the ground multiplet: a $Gamma_6$ ground state, with the two $Gamma_7$ states at 2 cm$^{-1}$, 95 cm$^{-1}$ and the other $Gamma_6$ state at 239 cm$^{-1}$. The $sim$2 cm$^{-1}$ separation between the CF ground and first excited states is in agreement with the previously proposed quasi-quartet CF ground state. The intensity of the lowest-energy CF transition remarkably increases on cooling, indicating a coupling of this CF excitation to the quadrupolar fluctuations above T$_0$. From symmetry analysis, we suggest that the FQ order has B$_{1g}$ symmetry. Moreover, temperature-dependent study of four Raman-active phonon modes shows that the intensities of the A$_{1g}$ and one E$_{g}$ modes increase significantly on cooling, which is explained by a near-resonant coupling between these two phonon modes and CF transitions.