Nuclear forces and hyperon forces are studied by lattice QCD. Simulations are performed with (almost) physical quark masses, $m_pi simeq 146$ MeV and $m_K simeq 525$ MeV, where $N_f=2+1$ nonperturbatively ${cal O}(a)$-improved Wilson quark action with stout smearing and Iwasaki gauge action are employed on the lattice of $(96a)^4 simeq (8.1mbox{fm})^4$ with $a^{-1} simeq 2.3$ GeV. In this report, we give the overview of the theoretical framework and present the numerical results for two-nucleon forces ($S=0$) and two-$Xi$ forces ($S=-4$). Central forces are studied in $^1S_0$ channel, and central and tensor forces are obtained in $^3S_1$-$^3D_1$ coupled channel analysis.