We extract the neutron electric dipole moment $vert vec{d}_Nvert$ within the lattice QCD formalism. We analyse one ensemble of $N_f=2+1+1$ twisted mass clover-improved fermions with lattice spacing of $a simeq 0.08 {rm fm}$ and physical values of the quark masses corresponding to a pion mass $m_{pi} simeq 139 {rm MeV}$. The neutron electric dipole moment is extracted by computing the $CP$-odd electromagnetic form factor $F_3(Q^2 to 0)$ through small $theta$-expansion of the action. This approach requires the calculation of the topological charge for which we employ a fermionic definition by means of spectral projectors while we also provide a comparison with the gluonic definition accompanied by the gradient flow. We show that using the topological charge from spectral projectors leads to absolute errors that are more than two times smaller than those provided when the field theoretic definition is employed. We find a value of $vert vec{d}_Nvert = 0.0009(24) theta e cdot {rm fm}$ when using the fermionic definition, which is statistically consistent with zero.