We study the spontaneous emission of an excited atom close to an optical nanofiber and the resulting scattering forces. For a suitably chosen orientation of the atomic dipole, the spontaneous emission pattern becomes asymmetric and a resonant Casimir--Polder force parallel to the fiber axis arises. For a simple model case, we show that the such a lateral force is due to the interaction of the circularly oscillating atomic dipole moment with its image inside the material. With the Casimir--Polder energy being constant in the lateral direction, the predicted lateral force does not derive from a potential in the usual way. Our results have implications for optical force measurements on a substrate as well as for laser cooling of atoms in nanophotonic traps.