High quality single crystals of BaFe$_{12}$O$_{19}$ were grown using the floating zone technique in flowing oxygen pressurized to 100 atm. Single crystal neutron diffraction was used to determine the nuclear and magnetic structure of BaFe$_{12}$O$_{19}$ at 4 K and 295 K. At both temperatures, there exist local electric dipoles formed by the off-mirror-plane displacements of magnetic Fe$^{3+}$ ions at the bipyramidal sites. The displacement at 4 K is about half of that at room temperature. The temperature dependence of the specific heat shows no anomaly associated with long range polar ordering in the temperature range from 1.90-300 K. The inverse dielectric permittivity, $1/varepsilon$, along the c-axis shows a $T^2$ temperature dependence between 10 K and 20 K, with a significantly reduced temperature dependence displayed below 10 K. Moreover, as the sample is cooled below 1.4 K there is an anomalous sharp upturn in $1/varepsilon$. These features resemble those of classic quantum paraelectrics such as SrTiO$_3$. The presence of the upturn in $1/varepsilon$ indicates that BaFe$_{12}$O$_{19}$ is a critical quantum paraelectric system with Fe$^{3+}$ ions involved in both magnetic and electric dipole formation.