Eternally inflating universes lead to an infinite number of Boltzmann brains but also an infinite number of ordinary observers. If we use the scale factor measure to regularize these infinities, the ordinary observers dominate the Boltzmann brains if the vacuum decay rate of each vacuum is larger than its Boltzmann brain nucleation rate. Here we point out that nucleation of small black holes should be counted in the vacuum decay rate, and this rate is always larger than the Boltzmann brain rate, if the minimum Boltzmann brain mass is more than the Planck mass. We also discuss nucleation of small, rapidly inflating regions, which may also have a higher rate than Boltzmann brains. This process also affects the distribution of the different vacua in eternal inflation.