In two-dimensional layered quantum materials, the stacking order of the layers determines both the crystalline symmetry and electronic properties such as the Berry curvature, topology and electron correlation. Electrical stimuli can influence quasiparticle interactions and the free-energy landscape, making it possible to dynamically modify the stacking order and reveal hidden structures that host different quantum properties. Here we demonstrate electrically driven stacking transitions that can be applied to design nonvolatile memory based on Berry curvature in few-layer WTe$_2$. The interplay of out-of-plane electric fields and electrostatic doping controls in-plane interlayer sliding and creates multiple polar and centrosymmetric stacking orders. In situ nonlinear Hall transport reveals such stacking rearrangements result in a layer-parity-selective Berry curvature memory in momentum space, where the sign reversal of the Berry curvature and its dipole only occurs in odd-layer crystals. Our findings open an avenue towards exploring coupling between topology, electron correlations, and ferroelectricity in hidden stacking orders and demonstrate a new low-energy-cost, electrically controlled topological memory in the atomically thin limit.