We show that Anderson localization in quasi-one dimensional conductors with ballistic electron dynamics, such as an array of ballistic chaotic cavities connected via ballistic contacts, can be understood in terms of classical electron trajectories only. At large length scales, an exponential proliferation of trajectories of nearly identical classical action generates an abundance of interference terms, which eventually leads to a suppression of transport coefficients. We quantitatively describe this mechanism in two different ways: the explicit description of transition probabilities in terms of interfering trajectories, and an hierarchical integration over fluctuations in the classical phase space of the array cavities.