Context. The process of mass accretion in the earliest phases of star formation is still not fully understood: Does the accretion rate smoothly decline with the age of the protostar or are there short, intermittent accretion bursts? Aims. We aim to explore whether or not the observed C$^{18}$O and N$_2$H$^+$ emission pattern towards the VeLLO IRAM 04191+1522 can be understood in the framework of a scenario where the emission is chemically tracing a past accretion burst. Methods.We used high-angular-resolution Plateau de Bure Interferometer (PdBI) observations of C$^{18}$O and N$_2$H$^+$ towards IRAM 04191+1522 that were obtained as part of the CALYPSO IRAM Large Program. We model these observations using a chemical code with a time-dependent physical structure coupled with a radiative transfer module, where we allow for variations in the source luminosity. Results. We find that the N$_2$H$^+$ line emission shows a central hole, while the C$^{18}$O emission is compact. The morphology of these two lines cannot be reproduced with a constant luminosity model based on the present-day internal luminosity (0.08 L$_{sun}$). However, the N$_2$H$^+$ peaks are consistent with a constant-luminosity model of 12 L$_{sun}$. Using a model with time-dependent temperature and density profiles, we show that the observed N$_2$H$^+$ peak emission could indeed be caused by a past accretion burst. Such a burst should have occurred a couple of hundred years ago. Conclusions. We suggest that an accretion burst occurred in IRAM 04191+1522 in the recent past. If such bursts are common and sufficiently long in VeLLOs, they could lead to higher accretion onto the central object than their luminosity suggests. For IRAM 04191 in particular, our results yield an estimated final mass of 0.2 - 0.25 M$_{sun}$ by the end of the Class 0 phase, which would make this object a low-mass star rather than a brown dwarf.