We report detailed temperature-dependent inelastic neutron scattering and ab-initio lattice dynamics investigation of magnetic perovskites YCrO3 and LaCrO3. The magnetic neutron scattering from the Cr ions exhibits significant changes with temperature and dominates at low momentum transfer regime. Ab-inito calculations performed including magnetic interactions show that the effect of magnetic interaction is very signicant on the low- as well as high-energy phonon modes. We have also shown that the inelastic neutron spectrum of YCrO3 mimics the magnon spectrum from a G-type antiferromagnetic system, which is consistent with previously reported magnetic structure in the compound. The ab-initio lattice dynamics calculations in both the compounds exhibit anisotropic thermal expansion behaviour in the orthorhombic structure and predict negative thermal expansion along the crystallographic a-axis at low temperatures. We identify the anharmonic phonon modes responsible for this anamolous behaviour in LaCrO3 involving low-energy La vibrations and distortions of the CrO6 octahedra.