We compute the dielectric response of glasses starting from a microscopic system-bath Hamiltonian of the Zwanzig-Caldeira-Leggett type and using an ansatz from kinetic theory for the memory function in the resulting Generalized Langevin Equation. The resulting framework requires the knowledge of the vibrational density of states (DOS) as input, that we take from numerical evaluation of a marginally-stable harmonic disordered lattice, featuring a strong boson peak (excess of soft modes over Debye $simomega_{p}^{2}$ law). The dielectric function calculated based on this ansatz is compared with experimental data for the paradigmatic case of glycerol at $Tlesssim T_{g}$. Good agreement is found for both the reactive (real part) of the response and for the $alpha$-relaxation peak in the imaginary part, with a significant improvement over earlier theoretical approaches, especially in the reactive modulus. On the low-frequency side of the $alpha$-peak, the fitting supports the presence of $sim omega_{p}^{4}$ modes at vanishing eigenfrequency as recently shown in [Phys. Rev. Lett. 117, 035501 (2016)]. $alpha$-wing asymmetry and stretched-exponential behaviour are recovered by our framework, which shows that these features are, to a large extent, caused by the soft boson-peak modes in the DOS.