In studies of cultural differentiation, the joint mechanisms of homophily and influence have been able to explain how distinct cultural groups can form. While these mechanisms normally lead to cultural convergence, increased levels of heterogeneity can allow them to produce global diversity. However, this emergent cultural diversity has proven to be unstable in the face of cultural drift- small errors or innovations that allow cultures to change from within. We develop a model of cultural differentiation that combines the traditional mechanisms of homophily and influence with a third mechanism of 2network homophily, in which network structure co-evolves with cultural interaction. We show that if social ties are allowed to change with cultural influence, a complex relationship between heterogeneity and cultural diversity is revealed, in which increased heterogeneity can reduce cultural group formation while simultaneously increasing social connectedness. Our results show that in certain regions of the parameter space these co-evolutionary dynamics can lead to patterns of cultural diversity that are stable in the presence of cultural drift.