Structural properties of SmRu$_4$P$_{12}$ in the anomalous magnetic ordered phase between $T^*sim 14 $ K and $T_{text{N}}=16.5$ K in magnetic fields has been studied by x-ray diffraction. Atomic displacements of Ru and P, reflecting the field-induced charge order of the $p$ electrons, have been deduced by analyzing the intensities of the forbidden Bragg peaks, assuming a cubic space group $Pmbar{3}$. Also, by utilizing high-resolution x-ray diffraction experiment, we observed a splitting of fundamental Bragg peaks, clarifying that the unit cell in the magnetic ordered phase is rhombohedral elongated along the $[1, 1, 1]$ axis. Responses of the rhombohedral domains to the magnetic field, which reflects the direction of the magnetic moment, is studied in detail.