We show that the anisotropies of the astrophysical stochastic gravitational wave background in the mHz band have a strong dependence on the modelling of galactic and sub-galactic physics. We explore a wide range of self-consistent astrophysical models for stellar evolution and for the distribution of orbital parameters, all calibrated such that they predict the same number of resolved mergers to fit the number of detections during LIGO/Virgo O1+O2 observations runs. We show that different physical choices for the process of black hole collapse and cut-off in the black hole mass distribution give fractional differences in the angular power spectrum of anisotropies up to 50% on all angular scales. We also point out that the astrophysical information which can be extracted from anisotropies is complementary to the isotropic background and individual mergers. These results underline the interest in the anisotropies of the stochastic gravitational wave background as a new and potentially rich field of research, at the cross-road between astrophysics and cosmology.