We indicated in our previous work that for QED the role of the scalar potential which appears at the loop level is much smaller than that of the vector potential and in fact negligible. But the situation is different for QCD, one reason is that the loop effects are more significant because $alpha_s$ is much larger than $alpha$, and secondly the non-perturbative QCD effects may induce a sizable scalar potential. In this work, we phenomenologically study the contribution of the scalar potential to the spectra of charmonia, bottomonia and $bbar c(bar b c)$ family. Taking into account both vector and scalar potentials, by fitting the well measured charmonia and bottomonia spectra, we re-fix the relevant parameters and test them by calculating other states of not only the charmonia, bottomonia, but also further the $bbar c$ family. We also consider the Lamb shift of the spectra.