Despite the rapid progress in understanding the first intrinsic magnetic topological insulator MnBi$_2$Te$_4$, its electronic structure remains a topic under debates. Here we perform a thorough spectroscopic investigation into the electronic structure of MnBi$_2$Te$_4$ via laser-based angle-resolved photoemission spectroscopy. Through quantitative analysis, we estimate an upper bound of 3 meV for the gap size of the topological surface state. Furthermore, our circular dichroism measurements reveal band chiralities for both the topological surface state and quasi-2D bands, which can be well reproduced in a band hybridization model. A numerical simulation of energy-momentum dispersions based on a four-band model with an additional step potential near the surface provides a promising explanation for the origin of the quasi-2D bands. Our study represents a solid step forward in reconciling the existing controversies in the electronic structure of MnBi$_2$Te$_4$, and provides an important framework to understand the electronic structures of other relevant topological materials MnBi$_{2n}$Te$_{3n+1}$.