We present the first lattice QCD determination of the $Lambda_b to Lambda^*(1520)$ vector, axial vector, and tensor form factors that are relevant for the rare decays $Lambda_b to Lambda^*(1520)ell^+ell^-$. The lattice calculation is performed in the $Lambda^*(1520)$ rest frame with nonzero $Lambda_b$ momenta, and is limited to the high-$q^2$ region. An interpolating field with covariant derivatives is used to obtain good overlap with the $Lambda^*(1520)$. The analysis treats the $Lambda^*(1520)$ as a stable particle, which is expected to be a reasonable approximation for this narrow resonance. A domain-wall action is used for the light and strange quarks, while the $b$ quark is implemented with an anisotropic clover action with coefficients tuned to produce the correct $B_s$ kinetic mass, rest mass, and hyperfine splitting. We use three different ensembles of lattice gauge-field configurations generated by the RBC and UKQCD collaborations, and perform extrapolations of the form factors to the continuum limit and physical pion mass. We give Standard-Model predictions for the $Lambda_b to Lambda^*(1520)ell^+ell^-$ differential branching fraction and angular observables in the high-$q^2$ region.