When granular systems are modeled by frictionless hard spheres, particle-particle collisions are considered as instantaneous events. This implies that while the velocities change according to the collision rule, the positions of the particles are the same before and after such an event. We show that depending on the material and system parameters, this assumption may fail. For the case of viscoelastic particles we present a universal condition which allows to assess whether the hard-sphere modeling and, thus, event-driven Molecular Dynamics simulations are justified.