Photon detection with quantum-level sensitivity is particularly challenging in the terahertz regime (0.1-10 THz), which contains ~98% of all the photons existing in the universe. Near-quantum-limited terahertz spectrometry has so far only been possible through the use of cryogenically cooled superconducting mixers as frequency downconverters. Here we introduce a spectrometry scheme that uses plasmonic photomixing for frequency downconversion to offer quantum-level sensitivities at room temperature for the first time. Frequency downconversion is achieved by mixing terahertz radiation and a heterodyning optical beam with a terahertz beat frequency in a plasmonics-enhanced semiconductor active region. We demonstrate spectrometer sensitivities down to 3 times the quantum-limit at room temperature. With a versatile design capable of broadband spectrometry, this plasmonic photomixer has broad applicability to quantum optics, chemical sensing, biological studies, medical diagnosis, high data-rate communication, as well as astronomy and atmospheric studies.