The electromagnetic form factors of the proton and the neutron are computed within lattice QCD using simulations with quarks masses fixed to their physical values. Both connected and disconnected contributions are computed. We analyze two new ensembles of $N_f = 2$ and $N_f = 2 + 1 + 1$ twisted mass clover-improved fermions and determine the proton and neutron form factors, the electric and magnetic radii, and the magnetic moments. We use several values of the sink-source time separation in the range of 1.0 fm to 1.6 fm to ensure ground state identification. Disconnected contributions are calculated to an unprecedented accuracy at the physical point. Although they constitute a small correction, they are non-negligible and contribute up to 15% for the case of the neutron electric charge radius.