In two and three spatial dimensions, the transverse response experienced by a charged particle on a lattice in a uniform magnetic field is proportional to a topological invariant, the first Chern number, characterizing the energy bands of the underlying Hofstadter Hamiltonian. In four dimensions, the transverse response is also quantized, and controlled by the second Chern number. These remarkable features solely arise from the magnetic translational symmetry. Here we show that the symmetries of the two-, three- and four-dimensional Hofstadter Hamiltonians may be encrypted in optical diffraction gratings, such that simple photonic experiments allow one to extract the first and the second Chern numbers of the whole energy spectra. This result is particularly remarkable in three and four dimensions, where complete topological characterizations have not yet been achieved experimentally. Side-by-side to the theoretical analysis, in this work we present the experimental study of optical gratings analogues of the two- and three-dimensional Hofstadter models.