FeVO$_4$ has been studied by heat capacity, magnetic susceptibility, electric polarization and single crystal neutron diffraction experiments. The triclinic crystal structure is made of emph{S}-shaped clusters of six Fe$^{3+}$ ions, linked by VO$_4^{3-}$ groups. Two long-range magnetic ordering transitions occur at T$_{N1}$=22K and T$_{N2}$=15K. Both magnetic structures are incommensurate. That stable below T$_{N1}$ is collinear with amplitude modulated moments whereas below T$_{N2}$ the arrangement is non-collinear with a helicoidal modulation. Below T$_{N2}$, fevo becomes weakly ferroelectric coincidentally with the loss of the collinearity of the magnetic structure. We conclude that fevo provides another example of frustrated spiral magnet similar to the classical TbMnO$_3$ compound. However, fevo has quenched orbital moments and a particular structure clarifying the respective role of anisotropy and magnetic frustration in this type of multiferroic materials.