Impurity effect is systematically studied in doped Haldane material Pb(Ni$_{1-x}$$M_x$)$_2$V$_2$O$_8$ ($M$ = Mn, Co, Cu, and Mg) by use of DC and AC susceptibility, and heat capacity measurements. The occurrence of three-dimensional ordered state is universally observed for all the impurities and the complete temperature -- concentration phase diagrams are obtained, which are qualitatively similar to that in other spin-gap materials. The unique feature is found in the drastic dependence of the transition temperatures on the species of the impurities. The consideration of effective Hamiltonian based on VBS model makes it clear that the ferromagnetic next-nearest-neighbor interaction and the antiferromagnetic nearest-neighbor interaction between impurity and edge spins play a key role in the unique feature.