The rheological response, in particular the non-linear response, to oscillatory shear is experimentally investigated in colloidal glasses. The glasses are highly concentrated binary hard-sphere mixtures with relatively large size disparities. For a size ratio of 0.2, a strong reduction of the normalized elastic moduli, the yield strain and stress and, for some samples, even melting of the glass to a fluid is observed upon addition of the second species. This is attributed to the more efficient packing, as indicated by the shift of random close packing to larger total volume fractions. This leads to an increase in free volume which favours cage deformations and hence a loosening of the cage. Cage deformations are also favoured by the structural heterogeneity introduced by the second species. For a limited parameter range, we furthermore found indications of two-step yielding, as has been reported previously for attractive glasses. In samples containing spheres with more comparable sizes, namely a size ratio of 0.38, the cage seems less distorted and structural heterogeneities on larger length scales seem to become important. The limited structural changes are reflected in only a small reduction of the moduli, yield strain and stress.