The physics governing electron acceleration by a relativistically intense laser are not confined to the critical density surface, they also pervade the sub-critical plasma in front of the target. Here, particles can gain many times the ponderomotive energy from the overlying laser, and strong fields can grow. Experiments using a high contrast laser and a prescribed laser pre-pulse demonstrate that development of the pre-plasma has an unexpectedly strong effect on the most energetic, super-ponderomotive electrons. Presented 2D particle-in-cell simulations reveal how strong, voluminous magnetic structures that evolve in the pre-plasma impact high energy electrons more significantly than low energy ones for longer pulse durations and how the common practice of tilting the target to a modest incidence angle can be enough to initiate strong deflection. The implications are that multiple angular spectral measurements are necessary to prevent misleading conclusions from past and future experiments.