We study the relation between the global topology of the Hofstadter butterfly of a multiband insulator and the topological invariants of the underlying Hamiltonian. The global topology of the butterfly, i.e., the displacement of the energy gaps as the magnetic field is varied by one flux quantum, is determined by the spectral flow of energy eigenstates crossing gaps as the field is tuned. We find that for each gap this spectral flow is equal to the topological invariant of the gap, i.e., the net number of edge modes traversing the gap. For periodically driven systems, our results apply to the spectrum of quasienergies. In this case, the spectral flow of the sum of all the quasienergies gives directly the Rudner invariant.