The off-diagonal Bethe ansatz method is generalized to the integrable model associated with the $sp(4)$ (or $C_2$) Lie algebra. By using the fusion technique, we obtain the complete operator product identities among the fused transfer matrices. These relations, together with some asymptotic behaviors and values of the transfer matrices at certain points, enable us to determine the eigenvalues of the transfer matrices completely. For the periodic boundary condition case, we recover the same $T-Q$ relations obtained via conventional Bethe ansatz methods previously, while for the off-diagonal boundary condition case, the eigenvalues are given in terms of inhomogeneous $T-Q$ relations, which could not be obtained by the conventional Bethe ansatz methods. The method developed in this paper can be directly generalized to generic $sp(2n)$ (i.e., $C_n$) integrable model.