P.B. Chakraborty {it et al.}, Phys. Rev. B {bf 70}, 144411 (2004)) study of the LiHoF$_4$ Ising magnetic material in an external transverse magnetic field $B_x$ show a discrepancy with the experimental results, even for small $B_x$ where quantum fluctuations are small. This discrepancy persists asymptotically close to the classical ferromagnet to paramagnet phase transition. In this paper, we numerically reinvestigate the temperature $T$, versus transverse field phase diagram of LiHoF$_4$ in the regime of weak $B_x$. In this regime, starting from an effective low-energy spin-1/2 description of LiHoF$_4$, we apply a cumulant expansion to derive an effective temperature-dependent classical Hamiltonian that incorporates perturbatively the small quantum fluctuations in the vicinity of the classical phase transition at $B_x=0$. Via this effective classical Hamiltonian, we study the $B_x-T$ phase diagram via classical Monte Carlo simulations. In particular, we investigate the influence on the phase diagram of various effects that may be at the source of the discrepancy between the previous QMC results and the experimental ones. For example, we consider two different ways of handling the long-range dipole-dipole interactions and explore how the $B_x-T$ phase diagram is modified when using different microscopic crystal field Hamiltonians. The main conclusion of our work is that we fully reproduce the previous QMC results at small $B_x$. Unfortunately, none of the modifications to the microscopic Hamiltonian that we explore are able to provide a $B_x-T$ phase diagram compatible with the experiments in the small semi-classical $B_x$ regime.