Very recently, an extremely bright fast radio burst (FRB) 200428 with two sub-millisecond pulses was discovered to come from the direction of the Galactic magnetar SGR 1935+2154, and an X-ray burst (XRB) counterpart was detected simultaneously. These observations favor magnetar-based interior-driven models. In this Letter, we propose a different model for FRB 200428 associated with an XRB from SGR 1935+2154, in which a magnetar with high proper velocity encounters an asteroid of mass $sim10^{20},$g. This infalling asteroid in the stellar gravitational field is first possibly disrupted tidally into a great number of fragments at radius $sim {rm a,,few}$ times $10^{10},$cm, and then slowed around the Alfv$acute{rm e}$n radius by an ultra-strong magnetic field and in the meantime two major fragments of mass $sim 10^{17},$g that cross magnetic field lines produce two pulses of FRB 200428. The whole asteroid is eventually accreted onto the poles along magnetic field lines, impacting the stellar surface, creating a photon-e$^pm$ pair fireball trapped initially in the stellar magnetosphere, and further leading to an XRB. We show that this gravitationally-powered model can interpret all of the observed features self-consistently.