If primordial black holes (PBHs) form directly from inhomogeneities in the early Universe, then the number in the mass range $10^5 -10^{12}M_{odot}$ is severely constrained by upper limits to the $mu$ distortion in the cosmic microwave background (CMB). This is because inhomogeneities on these scales will be dissipated by Silk damping in the redshift interval $5times 10^4lesssim zlesssim2times 10^6$. If the primordial fluctuations on a given mass scale have a Gaussian distribution and PBHs form on the high-$sigma$ tail, as in the simplest scenarios, then the $mu$ constraints exclude PBHs in this mass range from playing any interesting cosmological role. Only if the fluctuations are highly non-Gaussian, or form through some mechanism unrelated to the primordial fluctuations, can this conclusion be obviated.