A clock comparison experiment, analyzing the ratio of spin precession frequencies of stored ultracold neutrons and $^{199}$Hg atoms is reported. %57 No daily variation of this ratio could be found, from which is set an upper limit on the Lorentz invariance violating cosmic anisotropy field $b_{bot} < 2 times 10^{-20} {rm eV}$ (95% C.L.). This is the first limit for the free neutron. This result is also interpreted as a direct limit on the gravitational dipole moment of the neutron $|g_n| < 0.3 $eV/$c^2$ m from a spin-dependent interaction with the Sun. Analyzing the gravitational interaction with the Earth, based on previous data, yields a more stringent limit $|g_n| < 3 times 10^{-4} $eV/$c^2 $m.