The elementary vortex pinning potential is studied in a chiral p-wave superconductor with a pairing d=z(k_x + i k_y) on the basis of the quasiclassical theory of superconductivity. An analytical investigation and numerical results are presented to show that the vortex pinning potential is dependent on whether the vorticity and chirality are parallel or antiparallel. Mutual cancellation of the vorticity and chirality around a vortex is physically crucial to the effect of the pinning center inside the vortex core.