Rare earth triangular lattice materials have been proposed as a good platform for the investigation of frustrated magnetic ground states. KErSe$_2$ with the delafossite structure, contains perfect two-dimensional Er$^{3+}$ triangular layers separated by potassium ions, realizing this ideal configuration and inviting study. Here we investigate the magnetism of KErSe$_2$ at miliKelvin temperatures by heat capacity and neutron powder diffraction. Heat capacity results reveal a magnetic transition at 0.2 K in zero applied field. This long-range order is suppressed by an applied magnetic field of 0.5 T below 0.08 K. Neutron powder diffraction suggests that the zero-field magnetic structure orders with $k=(frac{1}{2},0,frac{1}{2})$ in a stripe spin structure. Unexpectedly, Er is found to have a reduced moment of 3.06(1) $mu_B$/Er in the ordered state and diffuse magnetic scattering, which originates at higher temperatures, is found to persist in the ordered state potentially indicating magnetic fluctuations. Neutron diffraction collected under an applied field shows a metamagnetic transition at $sim$ 0.5 T to ferromagnetic order with $k$=(0,0,0) and two possible structures, which are likely dependent on the applied field direction. First principle calculations show that the zero field stripe spin structure can be explained by the first, second and third neighbor couplings in the antiferromagnetic triangular lattice.