Two-dimensional triangular-lattice materials with spin-1/2 are perfect platforms for investigating quantum frustrated physics with spin fluctuations. Here we report the structure, magnetization, heat capacity and inelastic neutron scattering (INS) results on cesium ytterbium diselenide, CsYbSe$_2$. There is no long-range magnetic order down to 0.4 K at zero field. The temperature dependent magnetization, $M(T)$, reveals an easy-plane magnetic anisotropy. A maximum is found in $M(T)$ around emph{T}$sim$1.5 K when magnetic field $H$ is applied in the $ab$ plane, indicating the short-range interaction. The low-temperature isothermal magnetization $M(H)$ shows a one-third plateau of the estimated saturation moment, that is characteristic of a two-dimensional frustrated triangular lattice. Heat capacity shows field-induced long-range magnetic order for both $H||c$ and $H||ab$ directions. The broad peak in heat capacity and highly damped INS magnetic excitation at $T$=2 K suggests strong spin fluctuations. The dispersive in-plane INS, centered at the (1/3 1/3 0) point, and the absence of dispersion along $c$ direction suggests 120$^{circ}$ non-collinear 2D-like spin correlations. All these results indicate that the two-dimensional frustrated material CsYbSe$_2$ can be in proximity to the triangular-lattice quantum spin liquid. We propose an experimental low-temperature $H$-$T$ phase diagram for CsYbSe$_2$.