Driving laser wakefield acceleration with extremely short, near single-cycle laser pulses is crucial to the realisation of an electron source that can operate at kHz-repetition rate while relying on modest laser energy. It is also interesting from a fundamental point of view, as the ponderomotive approximation is no longer valid for such short pulses. Through particle-in-cell simulations, we show how the plasma response becomes asymmetric in the plane of laser polarization, and dependent on the carrier-envelope phase (CEP) of the laser pulse. For the case of self-injection, this in turn strongly affects the initial conditions of injected electrons, causing collective betatron oscillations of the electron beam. As a result, the electron beam pointing, electron energy spectrum and the direction of emitted betatron radiation become CEP-dependent. For injection in a density gradient the effect on beam pointing is reduced and the electron energy spectrum is CEP-independent, as electron injection is mostly longitudinal and mainly determined by the density gradient. Our results highlight the importance of controlling the CEP in this regime for producing stable and reproducible relativistic electron beams and identify how CEP effects may be observed in experiments. In the future, CEP control may become an additional tool to control the energy spectrum or pointing of the accelerated electron beam.