We report a systematic study of Sm valence in the prototypical intermediate valence compound SmB$_6$. Sm mean valence, $v_{rm Sm}$, was measured by X-ray absorption spectroscopy as functions of pressure ($1<P<13$ GPa) and temperature ($3<T<300$ K). Pressure induced magnetic order (MO) was detected above $P_c = 10$ GPa by resistivity measurements. A shift toward localized $4f$ state with increasing $P$ and/or $T$ is evident from an increase in $v_{rm Sm}$. However $v_{rm Sm}$ at $P_c$ is anomalously far below 3, which differs from the general case of nonmagnetic-magnetic transition in Yb and Ce compounds. From the $T$ dependence of $v_{rm Sm}(P,T)$, we found that $v_{rm Sm}(P,T)$ consists of two different characteristic components: one is associated with low-energy electronic correlations involving Kondo like behavior, and the other with high-energy valence fluctuations.