A systematic study of hole compensation effect on magnetic properties, which is controlled by defect compensation through ion irradiation, in (Ga,Mn)As, (In,Mn)As and (Ga,Mn)P is presented in this work. In all materials, both Curie temperature and magnetization decrease upon increasing the hole compensation, confirming the description of hole mediated ferromagnetism according to the p-d Zener model. The material dependence of Curie temperature and magnetization versus hole compensation reveals that the manipulation of magnetic properties in III-Mn-V dilute ferromagnetic semiconductors by ion irradiation is strongly influenced by the energy level location of the produced defect relative to the band edges in semiconductors.