We utilize the experimentally known difference of the $Lambda$ separation energies of the mirror hypernuclei ${^4_Lambda rm He}$ and ${^4_Lambda rm H}$ to constrain the $Lambda$-neutron interaction. We include the leading charge-symmetry breaking (CSB) interaction into our hyperon-nucleon interaction derived within chiral effective field theory at next-to-leading order. In particular, we determine the strength of the two arising CSB contact terms by a fit to the differences of the separation energies of these hypernuclei in the $0^+$ and $1^+$ states, respectively. By construction, the resulting interaction describes all low energy hyperon-nucleon scattering data, the hypertriton and the CSB in ${^4_Lambda rm He}$-${^4_Lambda rm H}$ accurately. This allows us to provide first predictions for the $Lambda$n scattering lengths, based solely on available hypernuclear data.