We consider the one-dimensional nonlinear Schrodinger equation with an attractive delta potential and mass-supercritical nonlinearity. This equation admits a one-parameter family of solitary wave solutions in both the focusing and defocusing cases. We establish asymptotic stability for all solitary waves satisfying a suitable spectral condition, namely, that the linearized operator around the solitary wave has a two-dimensional generalized kernel and no other eigenvalues or resonances. In particular, we extend our previous result beyond the regime of small solitary waves and extend the results of Fukuizumi-Ohta-Ozawa and Kaminaga-Ohta from orbital to asymptotic stability for a suitable family of solitary waves.