The connection between dark matter halos and galactic baryons is often not well-constrained nor well-resolved in cosmological hydrodynamical simulations. Thus, Halo Occupation Distribution (HOD) models that assign galaxies to halos based on halo mass are frequently used to interpret clustering observations, even though it is well-known that the assembly history of dark matter halos is related to their clustering. In this paper we use high-resolution hydrodynamical cosmological simulations to compare the halo and stellar mass growth of galaxies in a large-scale overdensity to those in a large-scale underdensity (on scales of about 20 Mpc). The simulation reproduces assembly bias, that halos have earlier formation times in overdense environments than in underdense regions. We find that the stellar mass to halo mass ratio is larger in overdense regions in central galaxies residing in halos with masses between 10$^{11}$-10$^{12.9}$ M$_{odot}$. When we force the local density (within 2 Mpc) at z=0 to be the same for galaxies in the large-scale over- and underdensities, we find the same results. We posit that this difference can be explained by a combination of earlier formation times, more interactions at early times with neighbors, and more filaments feeding galaxies in overdense regions. This result puts the standard practice of assigning stellar mass to halos based only on their mass, rather than considering their larger environment, into question.