We study the magnetic properties of the adatom systems on a semiconductor surface Si(111):{C,Si,Sn,Pb} - ($sqrt{3} times sqrt{3}$). On the basis of all-electron density functional theory calculations we construct effective low-energy models taking into account spin-orbit coupling and electronic correlations. In the ground state the surface nanostructures are found to be insulators with the non-collinear 120$^{circ}$ Neel (for C, Si, Sn monolayer coverages) and 120$^{circ}$ row-wise (for Pb adatom) antiferromagnetic orderings. The corresponding spin Hamiltonians with anisotropic exchange interactions are derived by means of the superexchange theory and the calculated Dzyaloshinskii-Moriya interactions are revealed to be very strong and compatible with the isotropic exchange couplings in the systems with Sn and Pb adatoms. To simulate the excited magnetic states we solve the constructed spin models by means of the Monte Carlo method. At low temperatures and zero magnetic field we observe complex spin spiral patterns in Sn/Si(111) and Pb/Si(111). On this basis the formation of antiferromagnetic skyrmion lattice states in adatom $sp$ electron systems in strong magnetic fields is discussed.