Quantum mechanics sets fundamental limits on how fast quantum states can be transformed in time. Two well-known quantum speed limits are the Mandelstam-Tamm (MT) and the Margolus-Levitin (ML) bounds, which relate the maximum speed of evolution to the systems energy uncertainty and mean energy, respectively. Here, we test concurrently both limits in a multi-level system by following the motion of a single atom in an optical trap using fast matter wave interferometry. Our data reveal two different regimes: one where the MT limit constrains the evolution at all times, and a second where a crossover to the ML limit is manifested at longer times. We take a geometric approach to quantify the deviation from the speed limit, measuring how much the matter waves quantum evolution deviates from the geodesic path in the Hilbert space of the multi-level system. Our results, establishing quantum speed limits beyond the simple two-level system, are important to understand the ultimate performance of quantum computing devices and related advanced quantum technologies.