We present radio observations of ultraluminous infrared galaxies (ULIRGs) using the Giant Metrewave Radio Telescope (GMRT) and combine them with archival multi-frequency observations to understand whether ULIRGs are the progenitors of the powerful radio loud galaxies in the local Universe. ULIRGs are characterized by large infrared luminosities ($L_{IR}>$10$^{12}$L$odot$), large dust masses ($sim10^{8}M_{odot}$) and vigorous star formation (star formation rates $sim$10-100 $M_{odot}~$yr$^{-1}$). Studies show that they represent the end stages of mergers of gas-rich spiral galaxies. Their luminosity can be due to both starburst activity and active galactic nuclei (AGN). We study a sample of 13 ULIRGs that have optically identified AGN characteristics with 1.28~GHz GMRT observations. Our aim is to resolve any core-jet structures or nuclear extensions and hence examine whether the ULIRGs are evolving into radio loud ellipticals. Our deep, low frequency observations show marginal extension for only one source. However, the integrated radio spectra of 9 ULIRGs show characteristics that are similar to that of GPS/CSS/CSO/young radio sources. The estimated spectral ages are 0.4 to 20 Myr and indicate that they are young radio sources and possible progenitors of radio galaxies. Hence, we conclude that although most ULIRGs do not show kpc scale extended radio emission associated with nuclear activity, their radio spectral energy distributions do show signatures of young radio galaxies.