We study the effects of non-magnetic impurity scattering on the Andreev bound states (ABS) in an isolated vortex in a two-dimensional chiral p-wave superconductor numerically. We incorporate the impurity scattering effects into the quasiclassical Eilenberger formulation through the self-consistent $t$-matrix approximation. Within this scheme, we calculate the local density of states (LDOS) around two types of vortices: parallel (anti-parallel) vortex where the phase winding of the pair-potential coming from vorticity and that coming from chirality have the same (opposite) sign. When the scattering phase-shift $delta_0$ of each impurity is small, we find that impurities affect differently low energy quasiparticle spectrum around the two types of vortex in a way similar to that in the Born limit ($delta_0rightarrow 0$). For a larger $delta_0(leq pi/2)$ however we find that ABS in the vortex is strongly suppressed by impurities for both types of vortex. We found that there are some correlations between the suppression of ABS near vortex cores and the low energy density of states due to impurity bands in the bulk.