The origin of intrinsic quantum criticality in the heavy-fermion superconductor $beta$-YbAlB$_4$ has been attributed to strong Yb valence fluctuations and its peculiar crystal structure. Here, we assess these contributions individually by studying the isostructural but fixed-valence compound $beta$-LuAlB$_4$. Quantum oscillation measurements and DFT calculations reveal a Fermi surface markedly different from that of $beta$-YbAlB$_4$, consistent with a `large Fermi surface there. We also find an unexpected in-plane anisotropy of the electronic structure, in contrast to the isotropic Kondo hybridization in $beta$-YbAlB$_4$.