Data from the Herschel Space Observatory have revealed an unusual elliptical galaxy, NGC 4125, which has strong and extended submillimeter emission from cold dust but only very strict upper limits to its CO and HI emission. Depending on the dust emissivity, the total dust mass is 2-5x10^6 Msun. While the neutral gas-to-dust mass ratio is extremely low (< 12-30), including the ionized gas traced by [CII] emission raises this limit to < 39-100. The dust emission follows a similar r^{1/4} profile to the stellar light and the dust to stellar mass ratio is towards the high end of what is found in nearby elliptical galaxies. We suggest that NGC 4125 is currently in an unusual phase where evolved stars produced in a merger-triggered burst of star formation are pumping large amounts of gas and dust into the interstellar medium. In this scenario, the low neutral gas-to-dust mass ratio is explained by the gas being heated to temperatures >= 10^4 K faster than the dust is evaporated. If galaxies like NGC 4125, where the far-infrared emission does not trace neutral gas in the usual manner, are common at higher redshift, this could have significant implications for our understanding of high redshift galaxies and galaxy evolution.