Josephson junctions (JJs) are ubiquitous superconducting devices, enabling high sensitivity magnetometers and voltage amplifiers, as well as forming the basis of high performance cryogenic computer and superconducting quantum computers. While JJ performance can be degraded by quasiparticles (QPs) formed from broken Cooper pairs, this phenomenon also opens opportunities to sensitively detect electromagnetic radiation. Here we demonstrate single near-infrared photon detection by coupling photons to the localized surface plasmons of a graphene-based JJ. Using the photon-induced switching statistics of the current-biased JJ, we reveal the critical role of QPs generated by the absorbed photon in the detection mechanism. The photon-sensitive JJ will enable a high-speed, low-power optical interconnect for future JJ-based computing architectures.