Vortex fiber nulling (VFN) is a method that may enable the detection and characterization of exoplanets at small angular separations (0.5-2 $lambda/D$) with ground- and space-based telescopes. Since the field of view is within the inner working angle of most coronagraphs, nulling accesses non-transiting planets that are otherwise too close to their star for spectral characterization by other means, thereby significantly increasing the number of known exoplanets available for direct spectroscopy in the near-infrared. Furthermore, VFN targets planets on closer-in orbits which tend to have more favorable planet-to-star flux ratios in reflected light. Here, we present the theory and applications of VFN, show that the optical performance is approximately equivalent for a variety of implementations and aperture shapes, and discuss the trade-offs between throughput and engineering requirements using numerical simulations. We compare vector and scalar approaches and, finally, show that beam shaping optics may be used to significantly improve the throughput for planet light. Based on theoretical performance, we estimate the number of known planets and theoretical exoEarths accessible with a VFN instrument linked to a high-resolution spectrograph on the future Thirty Meter Telescope.