The interplay of charge, spin, orbital and lattice degrees of freedom has recently received great interest due to its potential to improve the magnetocaloric effect (MCE) for the purpose of magnetic cooling applications. Here we propose a new mechanism for a giant inverse MCE in rare-earth tetraborides, especially for Ho1-xDyxB4 (x = 0.0, 0.5, and 1.0). For x = 0.0, 0.5, and 1.0, the maximum entropy changes of the giant inverse MCE are found to be 22.7 J/kgK, 19.6 J/kgK, and 19.0 J/kgK with critical fields of 25 kOe, 40 kOe, and 50 kOe, respectively. It is remarkable that such a giant MCE is realized, even when applying a low magnetic field, which enables a field-tuned entropy change and brings about a significant advantage for several applications. For all compounds, we have systematically studied how the entropy changes as a function of the field and temperature and investigated their correlation with consecutive double transitions, i.e., the magnetic dipolar order at T = TN and the quadrupolar order at T = TQ (TQ < TN). We found that the maximum entropy change occurs at T = TQ and the critical field associated with the meta-magnetic transition, which is in good agreement with the experimental data. Thus, we elucidate that this unique behaviour is attributed to the strong coupling between magnetic dipoles and quadrupoles in the presence of strong spin-orbit coupling and geometric frustration. Our work offers new insights into both the academic interest of multipolar degrees of freedom in magnetic materials and the discovery of giant MCE with various applications for magnetic cooling systems.