We compare the trends on the strength of electronic correlations across the different phases of elemental Pu focusing on its site and orbital dependence, using a combination of density functional theory (DFT) and dynamical mean field theory (DMFT) calculations within the vertex corrected one crossing approximation. We find that Pu-5$f$ states are more correlated in $delta$-Pu, followed by some crystallographic sites in $alpha$ and $beta$ phases. In addition, we observe that Pu-5$f_{5/2}$ and Pu-5$f_{7/2}$ orbital differentiation is a general feature of this material, as is site differentiation in the low symmetry phases. The Pu-5$f_{5/2}$ states show Fermi liquid like behavior whereas the Pu-5$f_{7/2}$ states remaining incoherent down to very low temperatures. We correlate the correlation strength in the different phases to their structure and the Pu-5$f$ occupancy of their crystallographic sites.