The ability to amplify light within silicon waveguides is central to the development of high-performance silicon photonic device technologies. To this end, the large optical nonlinearities made possible through stimulated Brillouin scattering offer a promising avenue for power-efficient all-silicon amplifiers, with recent demonstrations producing several dB of net amplification. However, scaling the degree of amplification to technologically compelling levels (>10 dB), necessary for everything from filtering to small signal detection, remains an important goal. Here, we significantly enhance the Brillouin amplification process by harnessing an inter-modal Brillouin interaction within a multi-spatial-mode silicon racetrack resonator. Using this approach, we demonstrate more than 20 dB of net Brillouin amplification in silicon, advancing state-of-the-art performance by a factor of 30. This degree of amplification is achieved with modest (~15 mW) continuous-wave pump powers and produces low out-of-band noise. Moreover, we show that this same system behaves as a unidirectional amplifier, providing more than 28 dB of optical nonreciprocity without insertion loss in an all-silicon platform. Building on these results, this device concept opens the door to new types of all-silicon injection-locked Brillouin lasers, high-performance photonic filters, and waveguide-compatible distributed optomechanical phenomena.