By LDA+U method with spin-orbit coupling (LDA+U+SO) the magnetic state and electronic structure have been investigated for plutonium in delta and alpha phases and for Pu compounds: PuN, PuCoGa5, PuRh2, PuSi2, PuTe, and PuSb. For metallic plutonium in both phases in agreement with experiment a nonmagnetic ground state was found with Pu ions in f^6 configuration with zero values of spin, orbital, and total moments. This result is determined by a strong spin-orbit coupling in 5f shell that gives in LDA calculation a pronounced splitting of 5f states on f^{5/2} and f^{7/2} subbands. A Fermi level is in a pseudogap between them, so that f^{5/2} subshell is already nearly completely filled with six electrons before Coulomb correlation effects were taken into account. The competition between spin-orbit coupling and exchange (Hund) interaction (favoring magnetic ground state) in 5f shell is so delicately balanced, that a small increase (less than 15%) of exchange interaction parameter value from J_H=0.48eV obtained in constrain LDA calculation would result in a magnetic ground state with nonzero spin and orbital moment values. For Pu compounds investigated in the present work, predominantly f^6 configuration with nonzero magnetic moments was found in PuCoGa5, PuSi2, and PuTe, while PuN, PuRh2, and PuSb have f^5 configuration with sizeable magnetic moment values. Whereas pure jj coupling scheme was found to be valid for metallic plutonium, intermediate coupling scheme is needed to describe 5f shell in Pu compounds. The results of our calculations show that both spin-orbit coupling and exchange interaction terms in the Hamiltonian must be treated in a general matrix form for Pu and its compounds.