Nitrogen vacancy (NV) centers in diamond have distinct promise as solid-state qubits. This is because of their large dipole moment, convenient level structure and very long room-temperature coherence times. In general, a combination of ion irradiation and subsequent annealing is used to create the centers, however for the rigorous demands of quantum computing all processes need to be optimized, and decoherence due to the residual damage caused by the implantation process itself must be mitigated. To that end we have studied photoluminescence (PL) from NV$^-$, NV$^0$ and GR1 centers formed by ion implantation of 2MeV He ions over a wide range of fluences. The sample was annealed at $600^{circ}$C to minimize residual vacancy diffusion, allowing for the concurrent analysis of PL from NV centers and irradiation induced vacancies (GR1). We find non-monotic PL intensities with increasing ion fluence, monotonic increasing PL in NV$^0$/NV$^-$ and GR1/(NV$^0$ + NV$^1$) ratios, and increasing inhomogeneous broadening of the zero-phonon lines with increasing ion fluence. All these results shed important light on the optimal formation conditions for NV qubits. We apply our findings to an off-resonant photonic quantum memory scheme using vibronic sidebands.