We derive a general relation between the fine structure splitting (FSS) and the exciton polarization angle of self-assembled quantum dots (QDs) under uniaxial stress. We show that the FSS lower bound under external stress can be predicted by the exciton polarization angle and FSS under zero stress. The critical stress can also be determined by monitoring the change in exciton polarization angle. We confirm the theory by performing atomistic pseudopotential calculations for the InAs/GaAs QDs. The work provides a deep insight into the dots asymmetry and their optical properties, and a useful guide in selecting QDs with smallest FSS which are crucial in entangled photon sources applications.