We report on the design, production, and performance of compact 40-cm$^3$ Time Projection Chambers (TPCs) that detect fast neutrons by measuring the three-dimensional (3D) ionization distribution of nuclear recoils in $^4$He:CO$_2$ gas at atmospheric pressure. We use these detectors to characterize the fast-neutron flux inside the Belle II detector at the SuperKEKB electron-positron collider in Tsukuba, Japan, where the primary design constraint is a small form factor. We find that the TPCs meet or exceed all design specifications, and are capable of measuring the 3D surface shape and charge density profile of ionization clouds from nuclear recoils and charged tracks in exquisite detail. Scaled-up detectors based on the detection principle demonstrated here may be suitable for directional dark matter searches, measurements of coherent neutrino-nucleus scattering, and other experiments requiring precise detection of neutrons or nuclear recoils.