Nd doped phosphate glasses have been studied before and after gamma irradiation in order to understand the effect of glass composition and radiation induced defects on the optical properties of glasses. UV, Vis absorption and photoluminescence spectra of these glasses are found strongly dependent on the composition of glass matrix, particularly on the ratio of oxygen (O) and neodymium (Nd) concentration obtained from energy dispersive X-ray spectroscopic (EDX) measurement. Gamma irradiation of glass modifies the transmission below 700 nm due to generation of some new absorption bands corresponding to different types of defects. Observations indicate toward possibility of change in the valence state of Nd3+ to Nd2+ and generation of oxygen vacancies in glass matrix. EDX and X-ray photoelectron spectroscopic (XPS) measurements indicate change in the composition of glasses particularly decrease in the relative concentration of oxygen in glass samples after gamma irradiation.