The small polaron, an one-dimensional lattice model of interacting spinless fermions, with generic non-diagonal boundary terms is studied by the off-diagonal Bethe ansatz method. The presence of the Grassmann valued non-diagonal boundary fields gives rise to a typical $U(1)$-symmetry-broken fermionic model. The exact spectra of the Hamiltonian and the associated Bethe ansatz equations are derived by constructing an inhomogeneous $T-Q$ relation.