Recently, several putative quantum spin liquid (QSL) states were discovered in ${tilde S} = 1/2$ rare-earth based triangular-lattice antiferromagnets (TLAF) with the delafossite structure. A way to clarify the origin of the QSL state in these systems is to identify ways to tune them from the putative QSL state towards long-range magnetic order. Here, we introduce the Ce-based TLAF KCeS$_2$ and show via low-temperature specific heat and $mu$SR investigations that it yields magnetic order below $T_{mathrm N} = 0.38$ K despite the same delafossite structure. We identify a well separated ${tilde S} = 1/2$ ground state for KCeS$_2$ from inelastic neutron scattering and embedded-cluster quantum chemical calculations. Magnetization and electron spin resonance measurements on single crystals indicate a strong easy-plane $g$~factor anisotropy, in agreement with the ab initio calculations. Finally, our specific-heat studies reveal an in-plane anisotropy of the magnetic field-temperature phase diagram which may indicate anisotropic magnetic interactions in KCeS$_2$.