We report an angle-resolved photoemission (ARPES) study of $beta$-YbAlB$_4$, which is known to harbor unconventional quantum criticality (QC) without any tuning. We directly observe a quasiparticle peak (QP), emerging from hybridization, characterized by a binding energy and an onset of coherence both at about 4 meV. This value conforms with a previously observed reduced Kondo scale at about 40 K. Consistency with an earlier study of carriers in $beta$-YbAlB$_4$ via the Hall effect strongly suggests that this QP is responsible for the QC in $beta$-YbAlB$_4$. A comparison with the sister polymorph $alpha$-YbAlB$_4$, which is not quantum critical at ambient pressure, further supports this result. Indeed, within the limitation of our instrumental resolution, our ARPES measurements do not show tangible sign of hybridization in this locally isomorphic system, while the conduction band we observe is essentially the same as in $beta$-YbAlB$_4$. We therefore claim that we identified by ARPES the carriers responsible for the QC in $beta$-YbAlB$_4$. The observed dispersion and the underlying hybridization of this QP are discussed in the context of existing theoretical models.