Nature of the field-induced charge ordered phase (phase II) of SmRu$_4$P$_{12}$ has been investigated by resonant x-ray diffraction (RXD) and polarized neutron diffraction (PND), focusing on the relationship between the atomic displacements and the antiferromagnetic (AFM) moments of Sm. From the analysis of the interference between the non-resonant Thomson scattering and the resonant magnetic scattering, combined with the spectral function obtained from x-ray magnetic circular dichroism, it is shown that the AFM moment of Sm prefers to be parallel to the field ($m_{text{AF}} parallel H$), giving rise to large and small moment sites around which the P$_{12}$ and Ru cage contract and expand, respectively. This is associated with the formation of the staggered ordering of the $Gamma_7$-like and $Gamma_8$-like crystal-field states, providing a strong piece of evidence for the charge order. PND was also performed to obtain complementary and unambiguous conclusion. In addition, isotropic and continuous nature of the phase II is demonstrated by the field-direction invariance of the interference spectrum in RXD. Crucial role of the $p$-$f$ hybridization is shown by resonant soft x-ray diffraction at the P $K$-edge ($1sleftrightarrow 3p$), where we detected a resonance due to the spin polarized $3p$ orbitals reflecting the AFM order of Sm.