Linear-dichroism is an important tool to characterize the transmission matrix and determine the crystal or orbital orientation in a material. In order to gain high resolution mapping of the transmission properties of such materials, we introduce the linear-dichroism scattering model in ptychographic imaging, and then develop an efficient two-stage reconstruction algorithm. Using proposed algorithm, the dichroic transmission matrix without an analyzer can be recovered by using ptychography measurements with as few as three different polarization angles, with the help of an empty region to remove phase ambiguities.