We have measured high-precision parallaxes for a large sample of candidate young (~10-100 Myr) and intermediate-age (~100-600 Myr) ultracool dwarfs, with spectral types ranging from M8 to T2.5. These objects are compelling benchmarks for substellar evolution and ultracool atmospheres at lower surface gravities (i.e., masses) than most of the field population. We find that the absolute magnitudes of our young sample can be systematically offset from ordinary (older) field dwarfs, with the young late-M objects being brighter and the young/dusty mid-L (L3-L6.5) objects being fainter, especially at J band. Thus, we conclude the underluminosity of the young planetary-mass companions HR 8799b and 2MASS J1207-39b compared to field dwarfs is also manifested in young free-floating brown dwarfs, though the effect is not as extreme. At the same time, some young objects over the full spectral type range of our sample are similar to field objects, and thus a simple correspondence between youth and magnitude offset relative to the field population appears to be lacking. Comparing the kinematics of our sample to nearby stellar associations and moving groups, we identify several new moving group members, including the first free-floating L dwarf in the AB Dor moving group, 2MASS J0355+11. Altogether, the effects of surface gravity (age) and dust content on the magnitudes and colors of substellar objects appear to be degenerate.