We report fluorescence investigations and Raman spectroscopy on colloidal nanodiamonds (NDs) obtained via bead assisted sonic disintegration (BASD) of a polycrystalline chemical vapor deposition film. The BASD NDs contain in situ created silicon vacancy (SiV) centers. Whereas many NDs exhibit emission from SiV ensembles, we also identify NDs featuring predominant emission from a single bright SiV center. We demonstrate oxidation of the NDs in air as a tool to optimize the crystalline quality of the NDs via removing damaged regions resulting in a reduced ensemble linewidth as well as single photon emission with increased purity. We furthermore investigate the temperature dependent zero-phonon-line fine-structure of a bright single SiV center as well as the polarization properties of its emission and absorption.