We test the history of structure formation from redshift 1 to today by matching galaxies from VIPERS and SDSS with dark matter haloes in the MultiDark SMDPL N-body simulation. We first show that the standard subhalo abundance matching (SHAM) recipe implemented with MultiDark characterizes the clustering of galaxies well both at redshift 0 for SDSS and at redshift 1 for VIPERS. This is an important validation of the SHAM model at high redshift. We then remap the simulation timesteps to test alternative growth histories and infer the growth index $gamma=0.6pm0.3$. This analysis demonstrates the power of using N-body simulations to forward model galaxy surveys for cosmological inference.