It has been clarified that bulk superconductivity in Fe$_{1+y}$Te$_{0.6}$Se$_{0.4}$ can be induced by annealing in an appropriate atmosphere to remove the harmful effects of excess iron. In order to clarify the details of the annealing process, we studied the changes in the physical properties and reaction products of Fe$_{1+y}$Te$_{0.6}$Se$_{0.4}$ annealed in pnictogen (P, As, Sb) atmospheres. Crystals annealed in a pnictogen atmosphere show bulk superconductivity and the values of $T_{c}$ and $J_{c}$ are about $14~$K and 2-4 $times 10{^5}~$A/cm$^2$ ($2~$K, self-field), respectively. It is also found that the reaction rate increases with the increase in the saturated vapor pressure of the pnictogen. Unexpectedly, the reaction products of Fe$_{1+y}$Te$_{0.6}$Se$_{0.4}$ after annealing in a P atmosphere mainly consist of FeTe$_2$. In addition, the amount of P required to obtain the optimal $T_{c}$ is much smaller than the amount of excess iron, which is similar to the case of oxygen annealing. P, oxygen, and to some extent As could serve as catalysts to form FeTe$_2$ to remove excess iron.