Half of the energy emitted by late-T- and Y-type brown dwarfs emerges at 3.5 < lambda um < 5.5. We present new L (3.43 < lambda um < 4.11) photometry obtained at the Gemini North telescope for nine late-T and Y dwarfs, and synthesize L from spectra for an additional two dwarfs. The targets include two binary systems which were imaged at a resolution of 0.25. One of these, WISEP J045853.90+643452.6AB, shows significant motion, and we present an astrometric analysis of the binary using Hubble Space Telescope, Keck Adaptive Optics, and Gemini images. We compare lambda ~4um observations to models, and find that the model fluxes are too low for brown dwarfs cooler than ~700K. The discrepancy increases with decreasing temperature, and is a factor of ~2 at T_eff=500K and ~4 at T_eff=400K. Warming the upper layers of a model atmosphere generates a spectrum closer to what is observed. The thermal structure of cool brown dwarf atmospheres above the radiative-convective boundary may not be adequately modelled using pure radiative equilibrium; instead heat may be introduced by thermochemical instabilities (previously suggested for the L- to T-type transition) or by breaking gravity waves (previously suggested for the solar system giant planets). One-dimensional models may not capture these atmospheres, which likely have both horizontal and vertical pressure/temperature variations.