In several Fe-based superconductors, slight $C_4$ symmetry breaking occurs at $T^*$, which is tens of Kelvin higher than the structural transition temperature $T_S$. In this hidden nematic state at $T_S<T<T^*$, the orthorhombicity is tiny [$phi=(a-b)/(a+b) ll 0.1$%], but clear evidences of bulk phase transition have been accumulated. To explain this long-standing mystery, we propose the emergence of antiferro-bond (AFB) order with the antiferro wavevector ${bf q}=(0,pi)$ at $T=T^*$, by which the characteristic phenomena below $T^*$ are satisfactorily explained. This AFB order originates from the inter-orbital nesting between the $d_{xy}$-orbital hole-pocket and the electron-pocket, and this inter-orbital bond order naturally explains the pseudogap, band-folding, and tiny nematicity that is linear in $T^*-T$. The hidden AFB order explains key experiments in both BaFe$_2$As$_2$ and NaFeAs, but it is not expected to occur in FeSe because of the absence of the $d_{xy}$-orbital hole-pocket.