We prove that a Shimura curve in the Siegel modular variety is not generically contained in the open Torelli locus as long as the rank of unitary part in its canonical Higgs bundle satisfies a numerical upper bound. As an application we show that the Coleman-Oort conjecture holds for Shimura curves associated to partial corestriction upon a suitable choice of parameters, which generalizes a construction due to Mumford.