A fundamental prediction of quantum mechanics is that there are random fluctuations everywhere in a vacuum because of the zero-point energy. Remarkably, quantum electromagnetic fluctuations can induce a measurable force between neutral objects, known as the Casimir effect, which has attracted broad interests. The Casimir effect can dominate the interaction between microstructures at small separations and has been utilized to realize nonlinear oscillation, quantum trapping, phonon transfer, and dissipation dilution. However, a non-reciprocal device based on quantum vacuum fluctuations remains an unexplored frontier. Here we report quantum vacuum mediated non-reciprocal energy transfer between two micromechanical oscillators. We modulate the Casimir interaction parametrically to realize strong coupling between two oscillators with different resonant frequencies. We engineer the systems spectrum to have an exceptional point in the parameter space and observe the asymmetric topological structure near it. By dynamically changing the parameters near the exceptional point and utilizing the non-adiabaticity of the process, we achieve non-reciprocal energy transfer with high contrast. Our work represents an important development in utilizing quantum vacuum fluctuations to regulate energy transfer at the nanoscale and build functional Casimir devices.