We consider universal shot noise in ballistic chaotic cavities from a semiclassical point of view and show that it is due to action correlations within certain groups of classical trajectories. Using quantum graphs as a model system we sum these trajectories analytically and find agreement with random-matrix theory. Unlike all action correlations which have been considered before, the correlations relevant for shot noise involve four trajectories and do not depend on the presence of any symmetry.