The Ulysses spacecraft is orbiting the Sun on a highly inclined ellipse ($ i = 79^{circ}$, perihelion distance 1.3 AU, aphelion distance 5.4 AU). Between January 1996 and December 1999 the spacecraft was beyond 3 AU from the Sun and crossed the ecliptic plane at aphelion in May 1998. In this four-year period 218 dust impacts were recorded with the dust detector on board. We publish and analyse the complete data set of both raw and reduced data for particles with masses $rm 10^{-16} g$ to $rm 10^{-8}$ g. Together with 1477 dust impacts recorded between launch of Ulysses and the end of 1995 published earlier cite{gruen1995c,krueger1999b}, a data set of 1695 dust impacts detected with the Ulysses sensor between October 1990 and December 1999 is now available. The impact rate measured between 1996 and 1999 was relatively constant with about 0.2 impacts per day. The impact direction of the majority of the impacts is compatible with particles of interstellar origin, the rest are most likely interplanetary particles. The observed impact rate is compared with a model for the flux of interstellar dust particles. The flux of particles several micrometers in size is compared with the measurements of the dust instruments on board Pioneer 10 and Pioneer 11 beyond 3 AU (Humes 1980, JGR, 85, 5841--5852, 1980). Between 3 and 5 AU, Pioneer results predict that Ulysses should have seen five times more ($rm sim 10 mu m$ sized) particles than actually detected.