We report the first search for the $K_L to pi^0 gamma$ decay, which is forbidden by Lorentz invariance, using the data from 2016 to 2018 at the J-PARC KOTO experiment. With a single event sensitivity of $(7.1pm 0.3_{rm stat.} pm 1.6_{rm syst.})times 10^{-8}$, no candidate event was observed in the signal region. The upper limit on the branching fraction was set to be $1.7times 10^{-7}$ at the 90% confidence level.