The orbital Feshbach resonance (OFR) is a novel scheme for magnetically tuning the interactions in closed-shell fermionic atoms. Remarkably, unlike the Feshbach resonances in alkali atoms, the open and closed channels of the OFR are only very weakly detuned in energy. This leads to a unique effect whereby a medium in the closed channel can Pauli block, or frustrate, the two-body scattering processes. Here, we theoretically investigate the impact of frustration in the few- and many-body limits of the experimentally accessible three-dimensional $^{173}$Yb system. We find that by adding a closed-channel atom to the two-body problem, the binding energy of the ground state is significantly suppressed, and by introducing a closed-channel Fermi sea to the many-body problem, we can drive the system towards weaker fermion pairing. These results are potentially relevant to superconductivity in solid-state multiband materials, as well as to the current and continuing exploration of unconventional Fermi-gas superfluids near the OFR.