Let $G$ be a finite group and let $p$ be a prime. Assume that there exists a prime $q$ dividing $|G|$ which does not divide the order of any $p$-local subgroup of $G$. If $G$ is $p$-solvable or $q$ divides $p-1$, then $G$ has a $p$-block of defect zero. The case $q=2$ is a well-known result by Brauer and Fowler.