The Atacama Cosmology Telescope Polarimeter (ACTPol) is a polarization sensitive upgrade to the Atacama Cosmology Telescope. Located at an elevation of 5190 m, ACTPol measures the Cosmic Microwave Background (CMB) temperature and polarization with arcminute-scale angular resolution. Calibration of the detector angles is a critical step in producing maps of the CMB polarization. Polarization angle offsets in the detector calibration can cause leakage in polarization from E to B modes and induce a spurious signal in the EB and TB cross correlations, which eliminates our ability to measure potential cosmological sources of EB and TB signals, such as cosmic birefringence. We present our optical modeling and measurements associated with calibrating the detector angles in ACTPol.