We investigated the temperature-dependent optical conductivity of NaV2O5 in the energy range 4 meV-4 eV. The intensities and the polarization dependence of the detected electronic excitations give a direct indication for a broken-parity electronic ground-state and for a non-centrosymmetric crystal structure of the system in the high-temperature phase. A direct two-magnon optical absorption process, proposed in this Letter, is in quantitative agreement with the optical data. By analyzing the optically allowed phonons at various temperatures above and below the phase transition, we conclude that a second-order change to a larger unit cell takes place below 34 K.