The switching dynamics of a single-domain BiFeO$_3$ thin films is investigated through combining the dynamics of polarization and Neel vector. The evolution of the ferroelectric polarization is described by the Landau-Khalatnikov (LK) equation, and the Landau-Lifshitz-Gilbert (LLG) equations for spins in two sublattices to model the time evolution of the antiferromagnetic order (Neel vector) in a G-type antiferromagnet. This work theoretically demonstrates that due to the rotation of the magnetic hard axis following the polarization reversal, the Neel vector can be switched by 180 degrees, while the weak magnetization can remain unchanged. The simulation results are consistent with the ab initio calculation, where the Neel vector rotates during polarization rotation, and also match our calculation of the dynamics of order parameter using Landau-Ginzburg theory. We also find that the switching time of the Neel vector is determined by the speed polarization switching and is predicted to be as short as 30 ps.