The recently discovered magnetization reversal driven solely by a femtosecond laser pulse has been shown to be a promising way to record information at record breaking speeds. Seeking to improve the recording density has raised intriguing fundamental question about the feasibility to combine the ultrafast temporal with sub-wavelength spatial resolution of magnetic recording. Here we report about the first experimental demonstration of sub-diffraction and sub-100 ps all-optical magnetic switching. Using computational methods we reveal the feasibility of sub-diffraction magnetic switching even for an unfocused incoming laser pulse. This effect is achieved via structuring the sample such that the laser pulse experiences a passive wavefront shaping as it couples and propagates inside the magnetic structure. Time-resolved studies with the help of photo-emission electron microscopy clearly reveal that the sub-wavelength switching with the help of the passive wave-front shaping can be pushed into sub-100 ps regime.