We make a proposal for calculating refined Gopakumar-Vafa numbers (GVN) on elliptically fibered Calabi-Yau 3-folds based on refined holomorphic anomaly equations. The key examples are smooth elliptic fibrations over (almost) Fano surfaces. We include a detailed review of existing mathematical methods towards defining and calculating the (unrefined) Gopakumar-Vafa invariants (GVI) and the GVNs on compact Calabi-Yau 3-folds using moduli of stable sheaves, in a language that should be accessible to physicists. In particular, we discuss the dependence of the GVNs on the complex structure moduli and on the choice of an orientation. We calculate the GVNs in many instances and compare the B-model predictions with the geometric calculations. We also derive the modular anomaly equations from the holomorphic anomaly equations by analyzing the quasi-modular properties of the propagators. We speculate about the physical relevance of the mathematical choices that can be made for the orientation.