We calculate the low-lying spectrum of charmed baryons in lattice QCD on the $32^3times64$, $N_f=2+1$ PACS-CS gauge configurations at the almost physical pion mass of $sim 156$ MeV/c$^2$. By employing a set of interpolating operators with different Dirac structures and quark-field smearings for the variational analysis, we extract the ground and first few excited states of the spin-$1/2$ and spin-$3/2$, singly-, doubly-, and triply-charmed baryons. Additionally, we study the $Xi_c$-$Xi_c^prime$ mixing and the operator dependence of the excited states in a variational approach. We identify several states that lie close to the experimentally observed excited states of the $Sigma_c$, $Xi_c$ and $Omega_c$ baryons, including some of the $Xi_c$ states recently reported by LHCb. Our results for the doubly- and triply-charmed baryons are suggestive for future experiments.