We present a ground-state cooling scheme for the mechanical degrees of freedom of mesoscopic magnetic particles levitated in low-frequency traps. Our method makes use of a binary sensor and suitably shaped pulses to perform weak, adaptive measurements on the position of the magnet. This allows us to precisely determine the position and momentum of the particle, transforming the initial high-entropy thermal state into a pure coherent state. The energy is then extracted by shifting the trap center. By delegating the task of energy extraction to a coherent displacement operation we overcome the limitations associated with cooling schemes that rely on the dissipation of a two-level system coupled to the oscillator. We numerically benchmark our protocol in realistic experimental conditions, including heating rates and imperfect readout fidelities, showing that it is well suited for magnetogravitational traps operating at cryogenic temperatures. Our results pave the way for ground-state cooling of micron-scale particles.