We use neutron powder diffraction to study on the non-superconducting phases of ThFeAsN$_{1-x}$O$_x$ with $x=0.15, 0.6$. In our previous results on the superconducting phase ThFeAsN with $T_c=$ 30 K, no magnetic transition is observed by cooling down to 6 K, and possible oxygen occupancy at the nitrogen site is shown in the refinement(H. C. Mao emph{et al.}, EPL, 117, 57005 (2017)). Here, in the oxygen doped system ThFeAsN$_{1-x}$O$_x$, two superconducting region ($0leqslant x leqslant 0.1$ and $0.25leqslant x leqslant 0.55$) have been identified by transport experiments (B. Z. Li emph{et al.}, J. Phys.: Condens. Matter 30, 255602 (2018)). However, within the resolution of our neutron powder diffraction experiment, neither the intermediate doping $x=0.15$ nor the heavily overdoped compound $x= 0.6$ shows any magnetic order from 300 K to 4 K. Therefore, while it shares the common phenomenon of two superconducting domes as most of 1111-type iron-based superconductors, the magnetically ordered parent compound may not exist in this nitride family.