We explore the possible regimes of decaying two-dimensional quantum turbulence, and elucidate the nature of spectral energy transport by introducing a dissipative point-vortex model with phenomenological vortex-sound interactions. The model is valid for a large system with weak dissipation, and also for systems with strong dissipation, and allows us to extract a meaningful and unambiguous spectral energy flux associated with quantum vortex motion. For weak dissipation and large system size we find a regime of hydrodynamic vortex turbulence in which energy is transported to large spatial scales, resembling the phenomenology of the transient inverse cascade observed in decaying turbulence in classical incompressible fluids. For strong dissipation the vortex dynamics are dominated by dipole recombination and exhibit no appreciable spectral transport of energy.