Infrastructure recovery time estimation is critical to disaster management and planning. Inspired by recent resilience planning initiatives, we consider a situation where experts are asked to estimate the time for different infrastructure systems to recover to certain functionality levels after a scenario hazard event. We propose a methodological framework to use expert-elicited data to estimate the expected recovery time curve of a particular infrastructure system. This framework uses the Gaussian process regression (GPR) to capture the experts estimation-uncertainty and satisfy known physical constraints of recovery processes. The framework is designed to find a balance between the data collection cost of expert elicitation and the prediction accuracy of GPR. We evaluate the framework on realistically simulated expert-elicited data concerning the two case study events, the 1995 Great Hanshin-Awaji Earthquake and the 2011 Great East Japan Earthquake.