Graphene-based photodetectors have shown responsivities up to 10$^8$A/W and photoconductive gains up to 10$^{8}$ electrons per photon. These photodetectors rely on a highly absorbing layer in close proximity of graphene, which induces a shift of the graphene chemical potential upon absorption, hence modifying its channel resistance. However, due to the semi-metallic nature of graphene, the readout requires dark currents of hundreds of $mu$A up to mA, leading to high power consumption needed for the device operation. Here we propose a novel approach for highly responsive graphene-based photodetectors with orders of magnitude lower dark current levels. A shift of the graphene chemical potential caused by light absorption in a layer of colloidal quantum dots, induces a variation of the current flowing across a metal-insulator-graphene diode structure. Owing to the low density of states of graphene near the neutrality point, the light-induced shift in chemical potential can be relatively large, dramatically changing the amount of current flowing across the insulating barrier, and giving rise to a novel type of gain mechanism. This readout requires dark currents of hundreds of nA up to few $mu$A, orders of magnitude lower than other graphene-based photodetectors, while keeping responsivities of $sim$70A/W in the infrared, almost two orders of magnitude higher compared to established germanium on silicon and indium gallium arsenide infrared photodetectors. This makes the device appealing for applications where high responsivity and low power consumption are required.