We present the single-epoch black hole mass (M$_{rm BH}$) calibrations based on the rest-frame UV and optical measurements of Mg II 2798AA and H$beta$ 4861AA lines and AGN continuum, using a sample of 52 moderate-luminosity AGNs at z$sim$0.4 and z$sim$0.6 with high-quality Keck spectra. We combine this sample with a large number of luminous AGNs from the Sloan Digital Sky Survey to increase the dynamic range for a better comparison of UV and optical velocity and luminosity measurements. With respect to the reference M$_{rm BH}$ based on the line dispersion of H$beta$ and continuum luminosity at 5100AA, we calibrate the UV and optical mass estimators, by determining the best-fit values of the coefficients in the mass equation. By investigating whether the UV estimators show systematic trend with Eddington ratio, FWHM of H$beta$, the Fe II strength, and the UV/optical slope, we find no significant bias except for the slope. By fitting the systematic difference of Mg II-based and H$beta$-based masses with the L$_{3000}$/L$_{5100}$ ratio, we provide a correction term as a function of the spectral index as $Delta$C = 0.24 (1+$alpha_{lambda}$) + 0.17, which can be added to the Mg II-based mass estimators if the spectral slope can be well determined. The derived UV mass estimators typically show $>$$sim$0.2 dex intrinsic scatter with respect to H$beta$-based M$_{rm BH}$, suggesting that the UV-based mass has an additional uncertainty of $sim$0.2 dex, even if high quality rest-frame UV spectra are available.