We present multilevel radiative transfer modeling of the scattering polarization observed in the solar O I infrared triplet around 777 nm. We demonstrate that the scattering polarization pattern observed on the solar disk forms in the chromosphere, far above the photospheric region where the bulk of the emergent intensity profiles originates. We study the sensitivity of the polarization pattern to the thermal structure of the solar atmosphere and to the presence of weak magnetic fields (0.01 - 100 G) through the Hanle effect, showing that the scattering polarization signals of the oxygen infrared triplet encode information on the magnetism of the solar chromosphere.