We investigate the dependence of galaxy clustering at $z sim 4 - 7$ on UV-luminosity and stellar mass. Our sample consists of $sim$ 10,000 Lyman-break galaxies (LBGs) in the XDF and CANDELS fields. As part of our analysis, the $M_star - M_{rm UV}$ relation is estimated for the sample, which is found to have a nearly linear slope of $dlog_{10} M_star / d M_{rm UV} sim 0.44$. We subsequently measure the angular correlation function and bias in different stellar mass and luminosity bins. We focus on comparing the clustering dependence on these two properties. While UV-luminosity is only related to recent starbursts of a galaxy, stellar mass reflects the integrated build-up of the whole star formation history, which should make it more tightly correlated with halo mass. Hence, the clustering segregation with stellar mass is expected to be larger than with luminosity. However, our measurements suggest that the segregation with luminosity is larger with $simeq 90%$ confidence (neglecting contributions from systematic errors). We compare this unexpected result with predictions from the textsc{Meraxes} semi-analytic galaxy formation model. Interestingly, the model reproduces the observed angular correlation functions, and also suggests stronger clustering segregation with luminosity. The comparison between our observations and the model provides evidence of multiple halo occupation in the small scale clustering.