Despite decades of research, the ultimate goal of nanotechnology--top-down manipulation of individual atoms--has been directly achieved with only one technique: scanning probe microscopy. In this Review, we demonstrate that scanning transmission electron microscopy (STEM) is emerging as an alternative method for the direct assembly of nanostructures, with possible applications in plasmonics, quantum technologies, and materials science. Atomically precise manipulation with STEM relies on recent advances in instrumentation that have enabled non-destructive atomic-resolution imaging at lower electron energies. While momentum transfer from highly energetic electrons often leads to atom ejection, interesting dynamics can be induced when the transferable kinetic energies are comparable to bond strengths in the material. Operating in this regime, very recent experiments have revealed the potential for single-atom manipulation using the Angstrom-sized electron beam. To truly enable control, however, it is vital to understand the relevant atomic-scale phenomena through accurate dynamical simulations. Although excellent agreement between experiment and theory for the specific case of atomic displacements from graphene has been recently achieved using density functional theory molecular dynamics, in many other cases quantitative accuracy remains a challenge. We provide a comprehensive reanalysis of available experimental data on beam-driven dynamics in light of the state-of-the-art in simulations, and identify important targets for improvement. Overall, the modern electron microscope has great potential to become an atom-scale fabrication platform, especially for covalently bonded 2D nanostructures. We review the developments that have made this possible, argue that graphene is an ideal starting material, and assess the main challenges moving forward.