We explore the boundaries of sine kernel universality for the eigenvalues of Gaussian perturbations of large deterministic Hermitian matrices. Equivalently, we study for deterministic initial data the time after which Dysons Brownian motion exhibits sine kernel correlations. We explicitly describe this time span in terms of the limiting density and rigidity of the initial points. Our main focus lies on cases where the initial density vanishes at an interior point of the support. We show that the time to reach universality becomes larger if the density vanishes faster or if the initial points show less rigidity.