In cite{APSIII} Atiyah, Patodi and Singer introduced spectral flow for elliptic operators on odd dimensional compact manifolds. They argued that it could be computed from the Fredholm index of an elliptic operator on a manifold of one higher dimension. A general proof of this fact was produced by Robbin-Salamon cite{RS95}. In cite{GLMST}, a start was made on extending these ideas to operators with some essential spectrum as occurs on non-compact manifolds. The new ingredient introduced there was to exploit scattering theory following the fundamental paper cite{Pu08}. These results do not apply to differential operators directly, only to pseudo-differential operators on manifolds, due to the restrictive assumption that spectral flow is considered between an operator and {its perturbation by a relatively trace-class operator}. In this paper we extend the main results of these earlier papers to spectral flow between an operator and a perturbation satisfying a higher $p^{th}$ Schatten class condition for $0leq p<infty$, thus allowing differential operators on manifolds of any dimension $d<p+1$. In fact our main result does not assume any ellipticity or Fredholm properties at all and proves an operator theoretic trace formula motivated by cite{BCPRSW, CGK16}. This leads us to introduce a notion of `generalised spectral flow for such paths and to investigate its properties.