We investigate the charmed baryon mass spectrum using the relativistic heavy quark action on 2+1 flavor PACS-CS configurations previously generated on $32^3 times 64$ lattice. The dynamical up-down and strange quark masses are tuned to their physical values, reweighted from those employed in the configuration generation. At the physical point, the inverse lattice spacing determined from the $Omega$ baryon mass gives $a^{-1}=2.194(10)$ GeV, and thus the spatial extent becomes $L = 32 a = 2.88(1)$ fm. Our results for the charmed baryon masses are consistent with experimental values, except for the mass of $Xi_{cc}$, which has been measured by only one experimental group so far and has not been confirmed yet by others. In addition, we report values of other doubly and triply charmed baryon masses, which have never been measured experimentally.