The insulator-to-metal transition (IMT) of the simple binary compound of vanadium dioxide VO$_2$ at $sim 340$ K has been puzzling since its discovery more than five decades ago. A wide variety of photon and electron probes have been applied in search
of a satisfactory microscopic mechanistic explanation. However, many of the conclusions drawn have implicitly assumed a {em homogeneous} material response. Here, we reveal inherently {em inhomogeneous} behavior in the study of the dynamics of individual VO$_2$ micro-crystals using a combination of femtosecond pump-probe microscopy with nano-IR imaging. The time scales of the photoinduced bandgap reorganization in the ultrafast IMT vary from $simeq 40 pm 8$ fs, i.e., shorter than a suggested phonon bottleneck, to $sim 200pm20$ fs, with an average value of $80 pm 25$ fs, similar to results from previous studies on polycrystalline thin films. The variation is uncorrelated with crystal size, orientation, transition temperature, and initial insulating phase. This together with details of the nano-domain behavior during the thermally-induced IMT suggests a significant sensitivity to local variations in, e.g., doping, defects, and strain of the microcrystals. The combination of results points to an electronic mechanism dominating the photoinduced IMT in VO$_2$, but also highlights the difficulty of deducing mechanistic information where the intrinsic response in correlated matter may not yet have been reached.
