We study the growth of entanglement in quantum systems with a conserved quantity exhibiting diffusive transport, focusing on how initial inhomogeneities are imprinted on the entropy. We propose a simple effective model, which generalizes the minimal cut picture of textit{Jonay et al.} in such a way that the `line tension of the cut depends on the local entropy density. In the case of noisy dynamics, this is described by a Kardar-Parisi-Zhang (KPZ) equation coupled to a diffusing field. We investigate the resulting dynamics and find that initial inhomogeneities of the conserved charge give rise to features in the entanglement profile, whose width and height both grow in time as $proptosqrt{t}$. In particular, for a domain wall quench, diffusion restricts entanglement growth to be $S_text{vN} lesssim sqrt{t}$. We find that for charge density wave initial states, these features in the entanglement profile are present even after the charge density has equilibrated. Our conclusions are supported by numerical results on random circuits and deterministic spin chains.