Quantum state smoothing is a technique for estimating the quantum state of a partially observed quantum system at time $tau$, conditioned on an entire observed measurement record (both before and after $tau$). However, this smoothing technique requires an observer (Alice, say) to know the nature of the measurement records that are unknown to her in order to characterize the possible true states for Bobs (say) systems. If Alice makes an incorrect assumption about the set of true states for Bobs system, she will obtain a smoothed state that is suboptimal, and, worse, may be unrealizable (not corresponding to a valid evolution for the true states) or even unphysical (not represented by a state matrix $rhogeq0$). In this paper, we review the historical background to quantum state smoothing, and list general criteria a smoothed quantum state should satisfy. Then we derive, for the case of linear Gaussian quantum systems, a necessary and sufficient constraint for realizability on the covariance matrix of the true state. Naturally, a realizable covariance of the true state guarantees a smoothed state which is physical. It might be thought that any putative true covariance which gives a physical smoothed state would be a realizable true covariance, but we show explicitly that this is not so. This underlines the importance of the realizabilty constraint.