We propose a scheme to directionally couple light into graphene plasmons by placing a graphene sheet on a magneto-optical substrate. When a magnetic field is applied parallel to the surface, the graphene plasmon dispersion relation becomes asymmetric in the forward and backward directions. It is possible to achieve unidirectional excitation of graphene plasmons with normally incident illumination by applying a grating to the substrate. The directionality can be actively controlled by electrically gating the graphene, or by varying the magnetic bias. This scheme may have applications in graphene-based opto-electronics and sensing.