We use data from the Sloan Digital Sky Survey and visual classifications of morphology from the Galaxy Zoo project to study black hole growth in the nearby Universe (z < 0.05) and to break down the AGN host galaxy population by color, stellar mass and morphology. We find that black hole growth at luminosities L_OIII >1E40 erg/s in early- and late-type galaxies is fundamentally different. AGN host galaxies as a population have a broad range of stellar masses (1E10-1E11 Msun), reside in the green valley of the color-mass diagram and their central black holes have median masses around 1E6.5 Msun. However, by comparing early- and late-type AGN host galaxies to their non-active counterparts, we find several key differences: in early-type galaxies, it is preferentially the galaxies with the least massive black holes that are growing, while late-type galaxies, it is preferentially the most massive}black holes that are growing. The duty cycle of AGN in early-type galaxies is strongly peaked in the green valley below the low-mass end (1E10 Msun) of the red sequence at stellar masses where there is a steady supply of blue cloud progenitors. The duty cycle of AGN in late-type galaxies on the other hand peaks in massive (1E11 Msun) green and red late-types which generally do not have a corresponding blue cloud population of similar mass. At high Eddington ratios (L/L_Edd > 0.1), the only population with a substantial fraction of AGN are the low-mass green valley early-type galaxies. Finally, the Milky Way likely resides in the sweet spot on the color-mass diagram where the AGN duty cycle of late-type galaxies is highest. We discuss the implications of these results for our understanding of the role of AGN in the evolution of galaxies