Magnetization measurements and time-of-flight neutron powder-diffraction studies on the high-temperature (300--980 K) magnetism and crystal structure (321--1200 K) of a pulverized YCrO$_3$ single crystal have been performed. Temperature-dependent inverse magnetic susceptibility coincides with a piecewise linear function with five regimes, with which we fit a Curie-Weiss law and calculate the frustration factor $f$. The fit results indicate a formation of magnetic polarons between 300 and 540 K and a very strong magnetic frustration. By including one factor $eta$ that represents the degree of spin interactions into the Brillouin function, we can fit well the applied-magnetic-field dependence of magnetization. No structural phase transition was observed from 321 to 1200 K. The average thermal expansions of lattice configurations (emph{a}, emph{b}, emph{c}, and emph{V}) obey well the Gr$ddot{textrm{u}}$neisen approximations with an anomaly appearing around 900 K, implying an isosymmetric structural phase transition, and display an anisotropic character along the crystallographic emph{a}, emph{b}, and emph{c} axes with the incompressibility $K^a_0 > K^c_0 > K^b_0$. It is interesting to find that at 321 K, the local distortion size $Delta$(O2) $approx$ 1.96$Delta$(O1) $approx$ 4.32$Delta$(Y) $approx$ 293.89$Delta$(Cr). Based on the refined Y-O and Cr-O bond lengths, we deduce the local distortion environments and modes of Y, Cr, O1, and O2 ions. Especially, the Y and O2 ions display obvious atomic displacement and charge subduction, which may shed light on the dielectric property of the YCrO$_3$ compound. Additionally, by comparing Kramers Mn$^{3+}$ with non-Kramers Cr$^{3+}$ ions, it is noted that being a Kramers or non-Kramers ion can strongly affect the local distortion size, whereas, it may not be able to change the detailed distortion mode.