The topological structure of the event horizon has been investigated in terms of the Morse theory. The elementary process of topological evolution can be understood as a handle attachment. It has been found that there are certain constraints on the nature of black hole topological evolution: (i) There are n kinds of handle attachments in (n+1)-dimensional black hole space-times. (ii) Handles are further classified as either of black or white type, and only black handles appear in real black hole space-times. (iii) The spatial section of an exterior of the black hole region is always connected. As a corollary, it is shown that the formation of a black hole with an S**(n-2) x S**1 horizon from that with an S**(n-1) horizon must be non-axisymmetric in asymptotically flat space-times.