Inelastic neutron scattering experiments on poly crystalline sample of heavy-fermion compound YbCo$_2$Zn$_{20}$ were carried out in order to obtain microscopic insights on the ground state and its magnetic field response. At zero field at 300 mK, inelastic response consists of two features: quasielastic scattering and a sharp peak at 0.6 meV. With increasing temperature, a broad peak comes up around 2.1 meV, whereas quasielastic response gets broader and the peak at 0.6 meV becomes unclear. By applying magnetic field, the quasielastic response exhibits significant broadening above 1 T, and the peak at 0.6 meV is obscure under fields. The peaks in inelastic spectra and its temperature variation can be ascribed to the suggested crystal-field model of ${{Gamma}_6}$ - ${{Gamma}_8}$ - ${{Gamma}_7}$ with the overall splitting of less than 3 meV. The observed quasielastic response and its rapid broadening with magnetic field indicates that the heavy-electron state arises from the ground state doublets, and are strongly suppressed by external field in YbCo$_2$Zn$_{20}$.