Inversion symmetry breaking is a ubiquitous concept in condensed-matter science. On the one hand, it is a prerequisite for many technologically relevant effects such as piezoelectricity, photovoltaic and nonlinear optical properties and spin-transport phenomena. On the other hand, it may determine abstract properties such as the electronic topology in quantum materials. Therefore, the creation of materials where inversion symmetry can be turned on or off by design may be the ultimate route towards controlling parity-related phenomena and functionalities. Here, we engineer the symmetry of ultrathin epitaxial oxide films by sub-unit-cell growth control. We reversibly activate and deactivate inversion symmetry in the layered hexagonal manganites, h-RMnO$_3$ with R = Y, Er, Tb. While an odd number of half-unit-cell layers exhibits a breaking of inversion symmetry through its arrangement of MnO$_5$ bipyramids, an even number of such half-unit-cell layers takes on a centrosymmetric structure. Here we control the resulting symmetry by tracking the growth in situ via optical second-harmonic generation. We furthermore demonstrate that our symmetry engineering works independent of the choice of R and even in heterostructures mixing constituents with different R in a two-dimensional growth mode. Symmetry engineering on the atomic level thus suggests a new platform for the controlled activation and deactivation of symmetry-governed functionalities in oxide-electronic epitaxial thin films.