Sr$_4$Ru$_3$O$_{10}$, the $n$ = 3 member of the Ruddlesden-Popper type ruthenate Sr$_{n+1}$Ru$_n$O$_{3n+1}$, is known to exhibit a peculiar metamagnetic transition in an in-plane magnetic field. However, the nature of both the temperature- and field-dependent phase transitions remains as a topic of debate. Here, we have investigated the magnetic transitions of Sr$_4$Ru$_3$O$_{10}$ via single-crystal neutron diffraction measurements. At zero field, we find that the system undergoes a ferromagnetic transition with both in-plane and out-of-plane magnetic components at $T_{c}$ ~ 100 K. Below $T^{*}$ ~ 50 K, the magnetic moments incline continuously toward the out-of-plane direction. At $T$ ~ 1.5 K, where the spins are nearly aligned along the $c$ axis, a spin reorientation occurs above a critical field $B_c$, giving rise to a spin component perpendicular to the plane defined by the field direction and the $c$ axis. We suggest that both the temperature- and field-driven spin reorientations are associated with a change in the magnetocrystalline anisotropy, which is strongly coupled to the lattice degrees of freedom. This study elucidates the long-standing puzzles on the zero-field magnetic orders of Sr$_4$Ru$_3$O$_{10}$ and provides new insights into the nature of the field-induced metamagnetic transition.