We report single-crystal neutron diffraction study of the magnetic structure of the multiferroic compound YbMnO$_3$, a member of the hexagonal manganite family, in zero-field and under a magnetic field applied along the $c$-axis. We propose a scenario for the zero-field magnetic ordering and for the field-induced magnetic reorientation of the Mn and of the two Yb on distinct crystallographic sites, compatible with the macroscopic measurements, as well as with previous powder neutron diffraction experiment and results from other techniques (optical second harmonic generation, Mossbauer spectroscopy). Our study should contribute in settling some debated issues about the magnetic properties of this material, as part of a broader investigation of the entire hexagonal RMnO$_3$ (R = Dy, Ho, Er, Tm, Yb, Lu, Y) family.