Two-dimensional electron gases (2DEGs) can form at the surface of oxides and semiconductors or in carefully designed quantum wells and interfaces. Depending on the shape of the confining potential, 2DEGs may experience a finite electric field, which gives rise to relativistic effects such as the Rashba spin-orbit coupling. Although the amplitude of this electric field can be modulated by an external gate voltage, which in turn tunes the 2DEG carrier density, sheet resistance and other related properties, this modulation is volatile. Here, we report the design of a ferroelectric 2DEG whose transport properties can be electrostatically switched in a non-volatile way. We generate a 2DEG by depositing a thin Al layer onto a SrTiO$_3$ single crystal in which 1 percent of Sr is substituted by Ca to make it ferroelectric. Signatures of the ferroelectric phase transition at 25 K are visible in the Raman response and in the temperature dependences of the carrier density and sheet resistance that shows a hysteretic dependence on electric field as a consequence of ferroelectricity. We suggest that this behavior may be extended to other oxide 2DEGs, leading to novel types of ferromagnet-free spintronic architectures.