Ductile fracture of metallic materials typically involves the elastoplastic deformation and associated damaging process. A nonlocal lattice particle method (LPM) is proposed to model this complex behavior. Recently, a distortional energy-based model is formulated into LPM to simulate the mixed linear hardening J2 plasticity. However, this model is based on the incremental updating algorithm which needs very small loading steps to get reasonable results. This is time-consuming and unstable for large systems. Therefore, in this paper, a stress-based return-mapping algorithm for simulating J2 plasticity is proposed to deal with these deficiencies. The material deterioration process is reformulated as a nonlocal damage evolution process. By incorporating the iterative solution procedure with dense-packing lattices, the damage-enhanced LPM framework is able to effectively reduce the lattice-dependency of crack grow analysis. The particle-size dependency of macroscopic mechanical responses is also handled properly by using the proposed nonlocal damage model. Several numerical examples are provided to show the ability of the new LPM framework to predict the elastoplastic behavior of engineering structures with/without damage and fracture.