Dense and narrow rings have been discovered recently around the small Centaur object Chariklo and the dwarf planet Haumea, while being suspected around the Centaur Chiron. They are the first rings observed in the Solar System elsewhere than around giant planets. Contrarily to the latters, gravitational fields of small bodies may exhibit large non-axisymmetric terms that create strong resonances between the spin of the object and the mean motion of rings particles. Here we show that modest topographic features or elongations of Chariklo and Haumea explain why their rings are relatively far away from the central body, when scaled to those of the giant planets. Lindblad-type resonances actually clear on decadal time-scales an initial collisional disk that straddles the corotation resonance (where the particles mean motion matches the spin rate of the body). The disk material inside the corotation radius migrates onto the body, while the material outside the corotation radius is pushed outside the 1/2 resonance, where the particles complete one revolution while the body completes two rotations. Consequently, the existence of rings around non-axisymmetric bodies requires that the 1/2 resonance resides inside the Roche limit of the body, favoring fast rotators for being surrounded by rings.