We present a detailed quantitative analysis of spectral correlations in the Sachdev-Ye-Kitaev (SYK) model. We find that the deviations from universal Random Matrix Theory (RMT) behavior are due to a small number of long-wavelength fluctuations from one realization of the ensemble to the next one. These modes can be parameterized effectively in terms of Q-Hermite orthogonal polynomials, the main contribution being the scale fluctuations for which we give a simple estimate. Our numerical results for $N=32$ show that only the lowest eight polynomials are needed to eliminate the nonuniversal part of the spectral fluctuations. The covariance matrix of the coefficients of this expansion is obtained analytically from low-order double-trace moments. We evaluate the covariance matrix of the first six moments and find that it agrees with the numerics. We also analyze the spectral correlations using a nonlinear $sigma$-model, which is derived through a Fierz transformation, and evaluate the one and two-point spectral correlator to two-loop order. The wide correlator is given by the sum of the universal RMT result and corrections whose lowest-order term corresponds to scale fluctuations. However, the loop expansion of the $sigma$-model results in an ill-behaved expansion of the resolvent, and it gives universal RMT fluctuations not only for $q=4$ but also for the $q=2$ SYK model while the correct result in this case should have been Poisson statistics. We analyze the number variance and spectral form factor for $N=32$ and $q=4$ numerically. We show that the quadratic deviation of the number variance for large energies appears as a peak for small times in the spectral form factor. After eliminating the long-wavelength fluctuations, we find quantitative agreement with RMT up to an exponentially large number of level spacings or exponentially short times, respectively.