We study the spectrum of anomalous dimensions of operators dual to giant graviton branes. The operators considered belong to the su$(2|3)$ sector of ${cal N}=4$ super Yang-Mills theory, have a bare dimension $sim N$ and are a linear combination of restricted Schur polynomials with $psim O(1)$ long rows or columns. In the same way that the operator mixing problem in the planar limit can be mapped to an integrable spin chain, we find that our problem maps to particles hopping on a lattice. The detailed form of the model is in precise agreement with the expected world volume dynamics of $p$ giant graviton branes, which is a U$(p)$ Yang-Mills theory. The lattice model we find has a number of noteworthy features. It is a lattice model with all-to-all sites interactions and quenched disorder.