The eventual detection of gravitational waves from core-collapse supernovae (CCSN) will help improve our current understanding of the explosion mechanism of massive stars. The stochastic nature of the late post-bounce gravitational wave signal due to the non-linear dynamics of the matter involved and the large number of degrees of freedom of the phenomenon make the source parameter inference problem very challenging. In this paper we take a step towards that goal and present a parameter estimation approach which is based on the gravitational waves associated with oscillations of proto-neutron stars (PNS). Numerical simulations of CCSN have shown that buoyancy-driven g-modes are responsible for a significant fraction of the gravitational wave signal and their time-frequency evolution is linked to the physical properties of the compact remnant through universal relations, as demonstrated in [1]. We use a set of 1D CCSN simulations to build a model that relates the evolution of the PNS properties with the frequency of the dominant g-mode, which is extracted from the gravitational-wave data using a new algorithm we have developed for our study. The model is used to infer the time evolution of a combination of the mass and the radius of the PNS. The performance of the method is estimated employing simulations of 2D CCSN waveforms covering a progenitor mass range between 11 and 40 solar masses and different equations of state. Considering signals embedded in Gaussian gravitational wave detector noise, we show that it is possible to infer PNS properties for a galactic source using Advanced LIGO and Advanced Virgo data at design sensitivities. Third generation detectors such as Einstein Telescope and Cosmic Explorer will allow to test distances of ${cal O}(100, {rm kpc})$.