Halo-based models have been successful in predicting the clustering of matter. However, the validity of the postulate that the clustering is fully determined by matter inside haloes remains largely untested, and it is not clear a priori whether non-virialised matter might contribute significantly to the non-linear clustering signal. Here, we investigate the contribution of haloes to the matter power spectrum as a function of both scale and halo mass by combining a set of cosmological N-body simulations to calculate the contributions of different spherical overdensity regions, Friends-of-Friends (FoF) groups and matter outside haloes to the power spectrum. We find that matter inside spherical overdensity regions of size R200,mean cannot account for all power for 1<k<100 h/Mpc, regardless of the minimum halo mass. At most, it accounts for 95% of the power (k>20 h/Mpc). For 2<k<10 h/Mpc, haloes with mass M200,mean<10^11 Msun/h contribute negligibly to the power spectrum, and our results appear to be converged with decreasing halo mass. When haloes are taken to be regions of size R200,crit, the amount of power unaccounted for is larger on all scales. Accounting also for matter inside FoF groups but outside R200,mean increases the contribution of halo matter on most scales probed here by 5-15%. Matter inside FoF groups with M200,mean>10^9 Msun/h accounts for essentially all power for 3<k<100 h/Mpc. We therefore expect halo models that ignore the contribution of matter outside R200,mean to overestimate the contribution of haloes of any mass to the power on small scales (k>1 h/Mpc).