Ultrafast control of light-matter interactions constitutes a crucial feature in view of new technological frontiers of information processing. However, conventional optical elements are either static or feature switching speeds that are extremely low with respect to the timescales at which it is possible to control light. Here, we exploit high-quality-factor engineered epsilon-near-zero (ENZ) modes of a metal-insulator-metal nanocavity to realize an all-optical ultrafast modulation of the reflectance of light at a tailored wavelength. Our approach is based on the presence of the two, spectrally separated, ENZ absorption resonances of the cavity. Optical pumping of the system at its high energy ENZ mode leads to a strong red-shift of the low energy mode because of the transient increase of the local dielectric function, which leads to a sub-3-ps control of the reflectance at a specific wavelength with a relative modulation depth approaching 120%.