We show that a class of compounds with $I$4/$mcm$ crystalline symmetry hosts three-dimensional semi-Dirac fermions. Unlike the known two-dimensional semi-Dirac points, the degeneracy of these three-dimensional semi-Dirac points is not lifted by spin-orbit coupling due to the protection by a nonsymmorphic symmetry -- screw rotation in the $a-b$ plane and a translation along the $c$ axis. This crystalline symmetry is found in tetragonal perovskite oxides, realizable in thin films by epitaxial strain that results in a$^0$a$^0$c$^-$-type octahedral rotation. Interestingly, with broken time-reversal symmetry, two pairs of Weyl points emerge from the semi-Dirac points within the Brillouin zone, and an additional lattice distortion leads to enhanced intrinsic anomalous Hall effect. We discuss possible fingerprints of this symmetry-protected band topology in electronic transport experiments.