The lower excitation spectrum of the nucleon and $Delta$ is calculated in a relativistic chiral quark model. Corrections to the baryon mass spectrum from the second order self-energy and exchange diagrams induced by pion and gluon fields are estimated in the field -theoretical framework. Convergent results for the self-energy terms are obtained when including the intermediate quark and antiquark states with a total momentum up to $j=25/2$. Relativistic one-meson and color-magnetic one-gluon exchange forces are shown to generate spin 0, 1, 2, etc. operators, which couple the lower and the upper components of the two interacting valence quarks and yield reasonable matrix elements for the lower excitation spectrum of the Nucleon and Delta. The only contribution to the ground state nucleon and $Delta$ comes from the spin 1 operators, which correspond to the exchanged pion or gluon in the l=1 orbit, thus indicating, that the both pion exchange and color-magnetic gluon exchange forces can contribute to the spin of baryons. Is is shown also that the contribution of the color-electric component of the gluon fields to the baryon spectrum is enormously large (more than 500 MeV with a value $alpha_s=0.65$) and one needs to restrict to very small values of the strong coupling constant or to exclude completely the gluon-loop corrections to the baryon spectrum. With this restriction, the calculated spectrum reproduces the main properties of the data, however needs further contribution from the two-pion exchange and instanton induced exchange (for the nucleon sector) forces in consistence with the realistic NN-interaction models.