With a uniform VLT SINFONI data set of nine targets, we have developed an empirical grid of J,H,K spectra of the atmospheres of objects estimated to have very low substellar masses of sim5-20 MJup and young ages of sim1-50 Myr. Most of the targets are companions, objects which are especially valuable for comparison with atmosphere and evolutionary models, as they present rare cases in which the age is accurately known from the primary. Based on the sample youth, all objects are expected to have low surface gravity, and this study investigates the critical early phases of the evolution of substellar objects. The spectra are compared with grids of five different theoretical atmosphere models. This analysis represents the first systematic model comparison with infrared spectra of young brown dwarfs. The fits to the full JHK spectra of each object result in a range of best fit effective temperatures of +/-150-300K whether or not the full model grid or a subset restricted to lower log(g) values is used. This effective temperature range is significantly larger than the uncertainty typically assigned when using a single model grid. Fits to a single wavelength band can vary by up to 1000K using the different models. Since the overall shape of these spectra is governed more by the temperature than surface gravity, unconstrained model fits did not find matches with low surface gravity or a trend in log(g) with age. This suggests that empirical comparison with spectra of unambiguously young objects targets (such as these SINFONI data) may be the most reliable method to search for indications of low surface gravity and youth. For two targets, the SINFONI data are a second epoch and the data show no variations in morphology over time. The analysis of two other targets, AB Pic B and CT Cha B, suggests that these objects may have lower temperatures, and consequently lower masses, than previously estimated.