Utilizing spin-polarized scanning tunneling microscopy and spectroscopy, we found coexistence of perpendicularly and in-plane magnetized cobalt nanoscale islands on the Ag(111) surface, and the relationship between the moire corrugation amplitude and the magnetization direction of the islands; the islands with the stronger moire corrugation show the perpendicular magnetization, and the ones with the weaker moire corrugation do the in-plane. Density functional theory calculations reproduce the relationship and explain the differences between the two types of the islands with an fcc stacking fault in the intrinsic hcp stacking of cobalt.