We introduce a textit{non-modal} analysis technique that characterizes the diffusion properties of spectral element methods for linear convection-diffusion systems. While strictly speaking only valid for linear problems, the analysis is devised so that it can give critical insights on two questions: (i) Why do spectral element methods suffer from stability issues in under-resolved computations of nonlinear problems? And, (ii) why do they successfully predict under-resolved turbulent flows even without a subgrid-scale model? The answer to these two questions can in turn provide crucial guidelines to construct more robust and accurate schemes for complex under-resolved flows, commonly found in industrial applications. For illustration purposes, this analysis technique is applied to the hybridized discontinuous Galerkin methods as representatives of spectral element methods. The effect of the polynomial order, the upwinding parameter and the Peclet number on the so-called textit{short-term diffusion} of the scheme are investigated. From a purely non-modal analysis point of view, polynomial orders between $2$ and $4$ with standard upwinding are well suited for under-resolved turbulence simulations. For lower polynomial orders, diffusion is introduced in scales that are much larger than the grid resolution. For higher polynomial orders, as well as for strong under/over-upwinding, robustness issues can be expected. The non-modal analysis results are then tested against under-resolved turbulence simulations of the Burgers, Euler and Navier-Stokes equations. While devised in the linear setting, our non-modal analysis succeeds to predict the behavior of the scheme in the nonlinear problems considered.