In this paper, we develop a sharp interface tumor growth model to study the effect of the tumor microenvironment using a complex far-field geometry that mimics a heterogeneous distribution of vasculature. Together with different nutrient uptake rates inside and outside the tumor, this introduces variability in spatial diffusion gradients. Linear stability analysis suggests that the uptake rate in the tumor microenvironment, together with chemotaxis, may induce unstable growth, especially when the nutrient gradients are large. We investigate the fully nonlinear dynamics using a spectrally accurate boundary integral method. Our nonlinear simulations reveal that vascular heterogeneity plays an important role in the development of morphological instabilities that range from fingering and chain-like morphologies to compact, plate-like shapes in two-dimensions.