Solid deuterium (sD_2) is used as an efficient converter to produce ultracold neutrons (UCN). It is known that the sD_2 must be sufficiently cold, of high purity and mostly in its ortho-state in order to guarantee long lifetimes of UCN in the solid from which they are extracted into vacuum. Also the UCN transparency of the bulk sD_2 material must be high because crystal inhomogeneities limit the mean free path for elastic scattering and reduce the extraction efficiency. Observations at the UCN sources at Paul Scherrer Institute and at Los Alamos National Laboratory consistently show a decrease of the UCN yield with time of operation after initial preparation or later treatment (`conditioning) of the sD_2. We show that, in addition to the quality of the bulk sD_2, the quality of its surface is essential. Our observations and simulations support the view that the surface is deteriorating due to a build-up of D_2 frost-layers under pulsed operation which leads to strong albedo reflections of UCN and subsequent loss. We report results of UCN yield measurements, temperature and pressure behavior of deuterium during source operation and conditioning, and UCN transport simulations. This, together with optical observations of sD_2 frost formation on initially transparent sD_2 in offline studies with pulsed heat input at the North Carolina State University UCN source results in a consistent description of the UCN yield decrease.