We report on a new measurement of the cosmic ray (CR) electron and positron spectra in the energy range of 20 MeV -- 1 GeV. The data were taken during the first flight of the balloon-borne spectrometer AESOP-Lite (Anti Electron Sub Orbital Payload), which was flown from Esrange, Sweden, to Ellesmere Island, Canada, in May 2018. The instrument accumulated over 130 hours of exposure at an average altitude of 3 g.cm$^{-2}$ of residual atmosphere. The experiment uses a gas Cherenkov detector and a magnetic spectrometer, consisting of a permanent dipole magnet and silicon strip detectors (SSDs), to identify particle type and measure the rigidity. Electrons and positrons were detected against a background of protons and atmospheric secondary particles. The primary cosmic ray spectra of electrons and positrons, as well as the re-entrant albedo fluxes, were extracted between 20 MeV -- 1 GeV during a positive solar magnetic polarity epoch. The positron fraction below 100 MeV appears flat, suggesting diffusion dominated solar modulation at low rigidity. The all-electron spectrum is presented and compared with models from a heliospheric numerical transport code.