The discovery of $Xi_{cc}^{++}$ by the LHCb Collaboration triggers predictions of more doubly charmed baryons. By taking into account both the $P$-wave excitations between the two charm quarks and the scattering of light pseudoscalar mesons off the ground state doubly charmed baryons, a set of negative-parity spin-1/2 doubly charmed baryons are predicted already from a unitarized version of leading order chiral perturbation theory. Moreover, employing heavy antiquark-diquark symmetry the relevant low-energy constants in the next-to-leading order are connected with those describing light pseudoscalar mesons scattering off charmed mesons, which have been well determined from lattice calculations and experimental data. Our calculations result in a spectrum richer than that of heavy mesons. We find two very narrow $J^P=1/2^-$ $Omega_{cc}^P$, which very likely decay into $Omega_{cc}pi^0$ breaking isospin symmetry. In the isospin-1/2 $Xi_{cc}^P$ sector, three states are predicted to exist below 4.2~GeV with the lowest one being narrow and the other two rather broad. We suggest to search for the $Xi_{cc}^{P}$ states in the $Xi_{cc}^{++}pi^-$ mode. Searching for them and their analogues are helpful to establish the hadron spectrum.