Four dimensional scanning transmission electron microscopy (4D STEM) records the scattering of electrons in a material in great detail. The benefits offered by 4D STEM are substantial, with the wealth of data it provides facilitating for instance high precision, high electron dose efficiency phase imaging via center of mass or ptychography based analysis. However the requirement for a 2D image of the scattering to be recorded at each probe position has long placed a severe bottleneck on the speed at which 4D STEM can be performed. Recent advances in camera technology have greatly reduced this bottleneck, with the detection efficiency of direct electron detectors being especially well suited to the technique. However even the fastest frame driven pixelated detectors still significantly limit the scan speed which can be used in 4D STEM, making the resulting data susceptible to drift and hampering its use for low dose beam sensitive applications. Here we report the development of the use of an event driven Timepix3 direct electron camera that allows us to overcome this bottleneck and achieve 4D STEM dwell times down to 100~ns; orders of magnitude faster than what has been possible with frame based readout. We characterise the detector for different acceleration voltages and show that the method is especially well suited for low dose imaging and promises rich datasets without compromising dwell time when compared to conventional STEM imaging.