We consider the problem of finding a dual formulation of gravity in the presence of non-trivial matter couplings. In the absence of matter a dual graviton can be introduced only for linearised gravitational interactions. We show that the coupling of linearised gravity to matter poses obstructions to the usual construction and comment on possible resolutions of this difficulty.