The search for the Majorana fermions in condensed matter physics has attracted much attention, partially because they may be used for the fault-tolerant quantum computation. It has been predicted that the Majorana zero mode may exist in the vortex core of topological superconductors. Recently, many iron-based superconductors are claimed to exhibit a topologically nontrivial surface state, including Fe(Te,Se). Some previous experiments through scanning tunneling microscopy (STM) have found zero-bias conductance peaks (ZBCP) within the vortex cores of Fe(Te,Se). However, our early experimental results have revealed the Caroli-de Gennes-Matricon (CdGM) discrete quantum levels in about 20% vortices. In many other vortices, we observed a dominant peak locating near zero bias. Here we show further study on the vortex core state of many more vortices in FeTe$_{0.55}$Se$_{0.45}$. In some vortices, if we take a certain criterion of bias voltage window near zero energy, we indeed see a zero energy mode. Some vortices exhibit zero energy bound state peaks with relatively symmetric background, which cannot be interpreted as the CdGM discrete states. The probability for vortices showing the ZBCP lowers down with the increase of magnetic field. Meanwhile it seems that the presence and absence of the ZBCP has no clear relationship with the Te/Se ratio on the surface. Temperature dependence of the spectra reveals that the ZBCP becomes weakened with increasing temperature and disappears at about 4 K. Our results provide a confirmed supplementary to the early claimed zero energy modes within the vortex cores of Fe(Te,Se). Detailed characterization of these zero energy modes versus magnetic field, temperature and spatial distribution of Te/Se will help to clarify its origin.