We report high-precision mass measurements of $^{50-55}$Sc isotopes performed at the LEBIT facility at NSCL and at the TITAN facility at TRIUMF. Our results provide a substantial reduction of their uncertainties and indicate significant deviations, up to 0.7 MeV, from the previously recommended mass values for $^{53-55}$Sc. The results of this work provide an important update to the description of emerging closed-shell phenomena at neutron numbers $N=32$ and $N=34$ above proton-magic $Z=20$. In particular, they finally enable a complete and precise characterization of the trends in ground state binding energies along the $N=32$ isotone, confirming that the empirical neutron shell gap energies peak at the doubly-magic $^{52}$Ca. Moreover, our data, combined with other recent measurements, does not support the existence of closed neutron shell in $^{55}$Sc at $N=34$. The results were compared to predictions from both emph{ab initio} and phenomenological nuclear theories, which all had success describing $N=32$ neutron shell gap energies but were highly disparate in the description of the $N=34$ isotone.