Color centers in diamond micro and nano structures are under investigation for a plethora of applications. However, obtaining high quality color centers in small structures is challenging, and little is known about how properties such as spin population lifetimes change during the transition from bulk to micro and nano structures. In this manuscript, we studied various ways to prepare diamond samples containing silicon vacancy centers and measured how population lifetimes of orbital states change in pillars as we varied their dimensions from approximately 1 $mu$m to 120 nm. We also researched the influence of the properties of the diamond substrate and the implantation and annealing methods on the silicon vacancy inhomogeneous linewidth and orbital lifetime. Our measurements show that nominally identical diamond samples can display significantly distinct inhomogeneous broadening. We observed weak indications that restricted vibrational modes in small structures may extend population lifetimes. However, imperfections in the crystal lattice or surface damage caused by etching reduce population lifetimes, especially in the smallest structures.