We prove the generic exclusion of certain Shimura varieties of unitary and orthogonal types from the Torelli locus. The proof relies on a slope inequality on surface fibration due to G. Xiao, and the main result implies that certain Shimura varieties only meet the Torelli locus in dimension zero.