We investigate the effects of point charge defects on the single particle electronic structure, emission energies, fine structure splitting and oscillator strengths of excitonic transitions in strained In$_{0.6}$Ga$_{0.4}$As/GaAs and strain-free GaAs/Al$_{0.3}$Ga$_{0.7}$As quantum dots. We find that the charged defects significantly modify the single particle electronic structure and excitonic spectra in both strained and strain-free structures. However, the excitonic fine structure splitting, polarization anisotropy and polarization direction in strained quantum dots remain nearly unaffected, while significant changes are observed for strain-free quantum dots.