The energy spacing between the ground-state spin doublet of $^4_Lambda $He(1$^+$,0$^+$) was determined to be $1406 pm 2 pm 2$ keV, by measuring $gamma$ rays for the $1^+ to 0^+$ transition with a high efficiency germanium detector array in coincidence with the $^4$He$(K^-,pi^-)$ $^4_Lambda $He reaction at J-PARC. In comparison to the corresponding energy spacing in the mirror hypernucleus $^4_Lambda $H, the present result clearly indicates the existence of charge symmetry breaking (CSB) in $Lambda N$ interaction. It is also found that the CSB effect is large in the $0^+$ ground state but is by one order of magnitude smaller in the $1^+$ excited state, demonstrating that the $Lambda N$ CSB interaction has spin dependence.