Spectroscopic factors to low-lying negative-parity states in $^{11}$Be extracted from the $^{12}$B($d$,$^3$He)$^{11}$Be proton-removal reaction are interpreted within the rotational model. Earlier predictions of the $p$-wave proton removal strengths in the strong coupling limit of the Nilsson model underestimated the spectroscopic factors to the $3/2^-_1$ and $5/2^-_1$ states and suggested that deviations in the $1^+$ ground state of the odd-odd $^{12}$B due to Coriolis coupling should be further explored. In this work we use the Particle Rotor Model to take into account these effects and obtain a good description of the level scheme in $^{11}$B, with a moderate $K$-mixing of the proton Nilsson levels [110]1/2 and [101]3/2. This mixing, present in the $1^+$ bandhead of $^{12}$B, is key to explaining the proton pickup data.