We study the effects of black hole dark matter on the dynamical evolution of stars in dwarf galaxies. We find that mass segregation leads to a depletion of stars in the center of dwarf galaxies and the appearance of a ring in the projected stellar surface density profile. Using Segue 1 as an example we show that current observations of the projected surface stellar density rule out at the 99.9% confidence level the possibility that more than 6% of the dark matter is composed of black holes with a mass of few tens of solar masses.