We present neutron diffraction analysis of BaFe$_2$(As$_{1-x}$P$_x$)$_2$ over a wide temperature (10 to 300 K) and compositional ($0.11 leq x leq 0.79$) range, including the normal state, the magnetically ordered state, and the superconducting state. The paramagnetic to spin-density wave and orthorhombic to tetragonal transitions are first order and coincident within the sensitivity of our measurements ($sim 0.5$ K). Extrapolation of the orthorhombic order parameter down to zero suggests that structural quantum criticality cannot exist at compositions higher than $x = 0.28$, which is much lower than values determined using other methods, but in good agreement with our observations of the actual phase stability range. The onset of spin-density wave order shows a stronger structural anomaly than the charge-doped system in the form of an enhancement of the $c/a$ ratio below the transition.