Why are the nuclei of some galaxies more active than others? If most galaxies harbor a central massive black hole, the main difference is probably in how well it is fueled by its surroundings. We investigate the hypothesis that such a difference can be seen in the detailed circumnuclear morphologies of galaxies using several quantitatively defined features, including bars, isophotal twists, boxy and disky isophotes, and strong non-axisymmetric features in unsharp masked images. These diagnostics are applied to 250 high-resolution images of galaxy centers obtained in the near-infrared with NICMOS on HST. To guard against the influence of possible biases and selection effects, we have carefully matched samples of Seyfert 1, Seyfert 2, LINER, starburst and normal galaxies in their basic properties, taking particular care to ensure that each was observed with a similar average scale (10-15 parsecs per pixel). Several morphological differences among our five different spectroscopic classifications emerge from the analysis. The HII/starburst galaxies show the strongest deviations from smooth elliptical isophotes, while the normal galaxies and LINERS have the least disturbed morphology. The Seyfert 2 galaxies have significantly more twisted isophotes than any other category, and the early-type Seyfert 2s are significantly more disturbed than the early-type Seyfert 1s. The morphological differences between Seyfert 1s and 2s suggest that more is at work than simply the viewing angle of the central engine. They may correspond to different evolutionary stages.