We report the first observation of two charmed strange baryons that decay into $Lambda_c^+ K^-pi^+$. The broader of the two states is measured to have a mass of $2978.5pm 2.1pm 2.0$ MeV/$c^2$ and a width of $43.5pm 7.5pm 7.0$ MeV/$c^2$. The mass and width of the narrow state are measured to be $3076.7pm 0.9pm 0.5$ MeV/$c^2$ and $6.2pm 1.2pm 0.8$ MeV/$c^2$, respectively. We also perform a search for the isospin partner states that decay into $Lambda_c^+ K_S^0pi^-$ and observe a significant signal at the mass of $3082.8pm 1.8pm 1.5$ MeV/$c^2$. The data used for this analysis was accumulated at or near the $Upsilon(4S)$ resonance, using the Belle detector at the $e^+ e^-$ asymmetric-energy collider KEKB. The integrated luminosity of the data sample used is $461.5 ~mathrm{fb}^{-1}$.