Linearly polarized spectra of far-infrared (IR) transmission in HoMn2O5 multiferroic single crystals have been studied in the frequency range between 8.5 and 105 cm-1 and for temperatures between 5 K and 300 K. Polarization of IR-active excitations depends on the crystallographic directions in HoMn2O5 and is sensitive to the magnetic phase transitions. We attribute some of the infrared-active excitations to electric-dipole transitions between ligand-field split states of Ho3+ ions. For light polarization along crystalline b-axis, the oscillator strength of electric dipoles at low frequencies (10.5, 13, and 18 cm-1) changes significantly at the commensurate-incommensurate antiferromagnetic phase transition at T3 = 19 K. This effect shows a strong correlation with the pronounced steps of the b-directional static dielectric function. We propose that the ligand field (LF) on Ho3+ connects the magnetism and dielectric properties of this compound through coupling with the Mn spin structure. We comment on the possibility for composite excitations of magnons and excited LF states.