Classical open systems with balanced gain and loss, i.e. parity-time ($mathcal{PT}$) symmetric systems, have attracted tremendous attention over the past decade. Their exotic properties arise from exceptional point (EP) degeneracies of non-Hermitian Hamiltonians that govern their dynamics. In recent years, increasingly sophisticated models of $mathcal{PT}$-symmetric systems with time-periodic (Floquet) driving, time-periodic gain and loss, and time-delayed coupling have been investigated, and such systems have been realized across numerous platforms comprising optics, acoustics, mechanical oscillators, optomechanics, and electrical circuits. Here, we introduce a $mathcal{PT}$-symmetric (balanced gain and loss) system with memory, and investigate its dynamics analytically and numerically. Our model consists of two coupled $LC$ oscillators with positive and negative resistance, respectively. We introduce memory by replacing either the resistor with a memristor, or the coupling inductor with a meminductor, and investigate the circuit energy dynamics as characterized by $mathcal{PT}$-symmetric or $mathcal{PT}$-symmetry broken phases. Due to the resulting nonlinearity, we find that energy dynamics depend on the sign and strength of initial voltages and currents, as well as the distribution of initial circuit energy across its different components. Surprisingly, at strong inputs, the system exhibits self-organized Floquet dynamics, including $mathcal{PT}$-symmetry broken phase at vanishingly small dissipation strength. Our results indicate that $mathcal{PT}$-symmetric systems with memory show a rich landscape.