The silicon-vacancy center in diamond offers attractive opportunities in quantum photonics due to its favorable optical properties and optically addressable electronic spin. Here, we combine both to achieve all-optical coherent control of its spin states. We utilize this method to explore spin dephasing effects in an impurity-rich sample beyond the limit of phonon-induced decoherence: Employing Ramsey and Hahn-echo techniques at 12mK base temperature we identify resonant coupling to a substitutional nitrogen spin bath as the limiting decoherence source for the electron spin.