We analyze the key origin of quantum valence criticality in the heavy electron metal $beta$-YbAlB$_4$ evidenced in the sister compound $alpha$-YbAl$_{0.986}$Fe$_{0.014}$B$_4$. By constructing a realistic canonical model for $beta$-YbAlB$_4$, we evaluate Coulomb repulsion between the 4f and 5d electrons at Yb $U_{rm fd}approx 6.2$ eV realizing the quantum critical point (QCP) of the Yb-valence transition. To reveal the Yb 5d contribution to the quantum critical state, we propose ultrasound measurement. We find that softening of elastic constants of not only the bulk modulus but also the shear moduli is caused by electric quadrupole fluctuations enhanced by critical 4f and 5d charge fluctuations for low temperatures at the valence QCP. Possible relevance of these results to $beta$-YbAlB$_4$ and also $alpha$-YbAl$_{1-x}$Fe$_x$B$_4$ is discussed.