In this work we analyse in detail the possibility of using small and intermediate-scale gravitational wave anisotropies to constrain the inflationary particle content. First, we develop a phenomenological approach focusing on anisotropies generated by primordial tensor-tensor-scalar and purely gravitational non-Gaussianities. We highlight the quantities that play a key role in determining the detectability of the signal. To amplify the power of anisotropies as a probe of early universe physics, we consider cross-correlations with CMB temperature anisotropies. We assess the size of the signal from inflationary interactions against so-called induced anisotropies. In order to arrive at realistic estimates, we obtain the projected constraints on the non-linear primordial parameter $F_{rm NL}$ for several upcoming gravitational wave probes in the presence of the astrophysical gravitational wave background. We further illustrate our findings by considering a concrete inflationary realisation and use it to underscore a few subtleties in the phenomenological analysis.