We report the results of the rapid follow-up observations of gamma-ray bursts (GRBs) detected by the Fermi satellite to search for associated fast radio bursts. The observations were conducted with the Australian Square Kilometre Array Pathfinder at frequencies from 1.2-1.4 GHz. A set of 20 bursts, of which four were short GRBs, were followed up with a typical latency of about one minute, for a duration of up to 11 hours after the burst. The data was searched using 4096 dispersion measure trials up to a maximum dispersion measure of 3763 pc cm$^{-3}$, and for pulse widths $w$ over a range of duration from 1.256 to 40.48 ms. No associated pulsed radio emission was observed above $26 {rm Jy ms} (w/1 {rm ms})^{-1/2}$ for any of the 20 GRBs.