The three-dimensional (3-D) reconstruction of nuclear recoils is of interest for directional detection of fast neutrons and for direction-sensitive searches for weakly interacting massive particles(WIMPs), which may constitute the Dark Matter of the universe. We demonstrate this capability with a miniature gas target Time Projection Chamber (TPC) where the drift charge is avalanche-multiplied with Gas Electron Multipliers (GEMs) and detected with the ATLAS FE-I3 Pixel Application Specific Integrated Circuit (ASIC). We report on performance characterization of the detector, including measurements of gain, gain resolution, point resolution, diffusion, angular resolution, and energy resolution with low-energy x-rays, cosmic rays, and alpha particles, using the gases Ar:CO$_2$ (70:30) and He:CO$_2$ (70:30) at atmospheric pressure. We discuss the implications for future, larger directional neutron and Dark Matter detectors. With an eye to designing and selecting components for these, we generalize our results into analytical expressions for detector performance whenever possible. We conclude by demonstrating the 3-D directional detection of a fast neutron source.