We study the Brownian motion of a charged test particle driven by quantum electromagnetic fluctuations in the vacuum region near a non-dispersive and non-absorbing dielectric half-space and calculate the mean squared fluctuations in the velocity of the test particle. Our results show that a nonzero susceptibility of the dielectrics has its imprints on the velocity dispersions of the test particles. The most noteworthy feature in sharp contrast to the case of an idealized perfectly conducting interface is that the velocity dispersions in the parallel directions are no longer negative and does not die off in time, suggesting that the potentially problematic negativeness of the dispersions in those directions in the case of perfect conductors is just a result of our idealization and does not occur for real material boundaries.