The quantum $tau_2$-model with generic site-dependent inhomogeneity and arbitrary boundary fields is studied via the off-diagonal Bethe Ansatz method. The eigenvalues of the corresponding transfer matrix are given in terms of an inhomogeneous T-Q relation, which is based on the operator product identities among the fused transfer matrices and the asymptotic behavior of the transfer matrices. Moreover, the associated Bethe Ansatz equations are also obtained.