We study the trend of the nuclear symmetry energy in relativistic mean-field models with appearance of the hyperon and quark degrees of freedom at high densities. On the pure hadron level, we focus on the role of $Lambda$ hyperons in influencing the symmetry energy both at given fractions and at charge and chemical equilibriums. The softening of the nuclear symmetry energy is observed with the inclusion of the $Lambda$ hyperons that suppresses the nucleon fraction. In the phase with the admixture of quarks and hadrons, the equation of state is established on the Gibbs conditions. With the increase of the quark volume fraction in denser and denser matter, the apparent nuclear symmetry energy decreases till to disappear. This softening would have associations with the observations which need detailed discriminations in dense matter with the admixture of new degrees of freedom created by heavy-ion collisions.