We consider a bipartite scenario where two parties hold ensembles of $1/2$-spins which can only be measured collectively. We give numerical arguments supporting the conjecture that in this scenario no Bell inequality can be violated for arbitrary numbers of spins if only first order moment observables are available. We then give a recipe to achieve a significant Bell violation with a split many-body system when this restriction is lifted. This highlights the strong requirements needed to detect bipartite quantum correlations in many-body systems device-independently.