The sample of dwarf galaxies with measured central black hole masses $M$ and velocity dispersions $sigma$ has recently doubled, and gives a close fit to the extrapolation of the $M propto sigma$ relation for more massive galaxies. We argue that this is difficult to reconcile with suggestions that the scaling relations between galaxies and their central black holes are simply a statistical consequence of assembly through repeated mergers. This predicts black hole masses significantly larger than those observed in dwarf galaxies unless the initial distribution of uncorrelated seed black hole and stellar masses is confined to much smaller masses than earlier assumed. It also predicts a noticeable flattening of the $M propto sigma$ relation for dwarfs, to $M propto sigma^2$ compared with the observed $M propto sigma^4$. In contrast black hole feedback predicts that black hole masses tend towards a universal $M propto sigma^4$ relation in all galaxies, and correctly gives the properties of powerful outflows recently observed in dwarf galaxies. These considerations emphasize once again that the fundamental physical black-hole -- galaxy scaling relation is between $M$ and $sigma$. The relation of $M$ to the bulge mass $M_b$ is acausal, and depends on the quite independent connection between $M_b$ and $sigma$ set by stellar feedback.