[Background:] The band structure of the negative-parity states of $^{24}$Mg has not yet been clarified. The $K^pi=0^-$, $K^pi=1^-$, and $K^pi=3^-$ bands have been suggested, but the assignments have been inconsistent between experiments and theories. [Purpose:] Negative-parity states of $^{24}$Mg are investigated by microscopic structure and reaction calculations via proton and alpha inelastic scattering to clarify the band assignment for the observed negative-parity spectra. [Method:] The structure of $^{24}$Mg was calculated using the antisymmetrized molecular dynamics~(AMD). Proton and alpha inelastic reactions were calculated using microscopic coupled-channel (MCC) calculations by folding the Melbourne $g$-matrix $NN$ interaction with the AMD densities of $^{24}$Mg. [Results:] The member states of the $K^pi=0^+$, $K^pi=2^+$, $K^pi=0^-$, $K^pi=1^-$, and $K^pi=3^-$ bands of $^{24}$Mg were obtained through the AMD result. In the MCC+AMD results for proton and alpha elastic and inelastic cross sections, reasonable agreements were obtained with existing data, except in the case of the $4^+_1$ state. [Conclusions:] The $3^-$ state of the $K^pi=3^-$ band and the $1^-$ and $3^-$ states of the $K^pi=0^-$ bands were assigned to the $3^-_1$(7.62 MeV), $1^-_1$(7.56 MeV), and $3^-_2$(8.36 MeV) states, respectively. The present AMD calculation is the first microscopic structure calculation to reproduce the energy ordering of the $K^pi=0^-$, $K^pi=1^-$, and $K^pi=3^-$ bands of $^{24}$Mg.