We propose a fast, scalable all-optical design for arbitrary two-qubit operations for defect qubits in diamond (NV centers) and in silicon carbide, which are promising candidates for room temperature quantum computing. The interaction between qubits is carried out by microcavity photons. The approach uses constructive interference from higher energy excited states activated by optical control. In this approach the cavity mode remains off-resonance with the directly accessible optical transitions used for initialization and readout. All quantum operations are controlled by near-resonant narrow-bandwidth optical pulses. We perform full quantum numerical modeling of the proposed gates and show that high-fidelity operations can be obtained with realistic parameters.