We present a study of the upper critical field, H$_{c2}$, of pristine and proton-irradiated RbEuFe$_4$As$_4$ crystals in pulsed magnetic fields of up to 65 T. The data for H$_{c2}$ reveal pronounced downwards curvature, particularly for the in-plane field orientation, and a superconducting anisotropy that decreases with decreasing temperature. These features are indicative of Pauli paramagnetic limiting. For the interpretation of these data, we use a model of a clean single-band superconductor with an open Fermi surface in the shape of a warped cylinder, which includes strong paramagnetic limiting. Fits to the data reveal that the in-plane upper critical field is Pauli paramagnetic limited, while the out-of-plane upper critical field is orbitally limited and that the orbital and paramagnetic fields have opposite anisotropies. A consequence of this particular combination is the unusual inversion of the anisotropy, $H_{c2}^{ab} < H_{c2}^c$, of the irradiated sample at temperatures below 10 K. The fits also yield an in-plane Maki parameter, $alpha_M^{110} approx$ 2.6, exceeding the critical value for the formation of the Fulde-Ferrell-Larkin-Ovchinnikov state. Nevertheless, the current measurements did not reveal direct evidence for the occurrence of this state.