We present a study of the morphology of the ejecta in Supernova 1987A based on images and spectra from the HST as well as integral field spectroscopy from VLT/SINFONI. The HST observations were obtained between 1994 - 2011 and primarily probe the outer hydrogen-rich zones of the ejecta. The SINFONI observations were obtained in 2005 and 2011 and instead probe the [Si I]/[Fe II] emission from the inner regions. We find a strong temporal evolution of the morphology in the HST images, from a roughly elliptical shape before ~5,000 days, to a more irregular, edge-brightened morphology thereafter. This transition is a natural consequence of the change in the dominant energy source powering the ejecta, from radioactive decay before ~5,000 days to X-ray input from the circumstellar interaction thereafter. The [Si I]/[Fe II] images display a more uniform morphology, which may be due to a remaining significant contribution from radioactivity in the inner ejecta and the higher abundance of these elements in the core. Both the H-alpha and the [Si I]/[Fe II] line profiles show that the ejecta are distributed fairly close to the plane of the inner circumstellar ring, which is assumed to define the rotational axis of the progenitor. The H-alpha emission extends to higher velocities than [Si I]/[Fe II] as expected. There is no clear symmetry axis for all the emission and we are unable to model the ejecta distribution with a simple ellipsoid model with a uniform distribution of dust. Instead, we find that the emission is concentrated to clumps and that the emission is distributed somewhat closer to the ring in the north than in the south. This north-south asymmetry may be partially explained by dust absorption. We compare our results with explosion models and find some qualitative agreement, but note that the observations show a higher degree of large-scale asymmetry.