We propose an original test of Lorentz invariance in the interaction between a particle spin and an electromagnetic field and report on a first measurement using ultracold neutrons. We used a high sensitivity neutron electric dipole moment (nEDM) spectrometer and searched for a direction dependence of a nEDM signal leading to a modulation of its magnitude at periods of 12 and 24 hours. We constrain such a modulation to $d_{12} < 15 times 10^{-25} e,{rm cm}$ and $d_{24} < 10 times 10^{-25} e,{rm cm}$ at 95~% C.L. The result translates into a limit on the energy scale for this type of Lorentz violation effect at the level of ${cal E}_{LV} > 10^{10}$~GeV.