We study the dynamics arising from a double quantum quench where the parameters of a given Hamiltonian are abruptly changed from being in an equilibrium phase A to a different phase B and back (A$to$B$to$A). As prototype models, we consider the (integrable) transverse field Ising as well as the (non-integrable) ANNNI model. The return amplitude features non-analyticities after the first quench through the equilibrium quantum critical point (A$to$B), which is routinely taken as a signature of passing through a so-called dynamical quantum phase transition. We demonstrate that non-analyticities after the second quench (B$to$A) can be avoided and reestablished in a recurring manner upon increasing the time $T$ spent in phase B. The system retains an infinite memory of its past state, and one has the intriguing opportunity to control at will whether or not dynamical quantum phase transitions appear after the second quench.