Coalescing binary black holes emit anisotropic gravitational radiation. This causes a net emission of linear momentum that produces a gradual acceleration of the source. As a result, the final remnant black hole acquires a characteristic velocity known as recoil velocity or gravitational kick. The symmetries of gravitational wave emission are reflected in the interactions of the gravitational wave modes emitted by the binary. In this work we make use of the rich information encoded in the higher-order modes of the gravitational wave emission to infer the component of the kick along the line-of-sight (or textit{radial kick}). We do this by performing parameter inference on simulated signals given by numerical relativity waveforms for non-spinning binaries using numerical relativity templates of aligned-spin (non-precessing) binary black holes. We find that for suitable sources, namely those with mass ratio $qgeq 2$ and total mass $M sim 100M_odot$, and for modest radial kicks of $120km/s$, the $90%$ credible intervals of our posterior probability distributions can exclude a zero kick at a signal-to-noise ratio of $15$; using a single Advanced LIGO detector working at its early sensitivity. The measurement of a non-zero radial kick component would provide the first observational signature of net transport of linear momentum by gravitational waves away from their source.