We present new evidence for a problem with cooling rates predicted by substellar evolutionary models that implies model-derived masses in the literature for brown dwarfs and directly imaged planets may be too high. Based on our dynamical mass for Gl 417BC (L4.5+L6) and a gyrochronology system age from its young, solar-type host star, commonly used models predict luminosities 0.2$-$0.4 dex lower than we observe. This corroborates a similar luminosity$-$age discrepancy identified in our previous work on the L4+L4 binary HD 130948BC, which coincidentally has nearly identical component masses ($approx$50$-$55 $M_{rm Jup}$) and age ($approx$800 Myr) as Gl 417BC. Such a luminosity offset would cause systematic errors of 15%$-$25% in model-derived masses at this age. After comparing different models, including cloudless models that should not be appropriate for mid-L dwarfs like Gl 417BC and HD 130948BC but actually match their luminosities better, we speculate the observed over-luminosity could be caused by opacity holes (i.e., patchy clouds) in these objects. Moreover, from hybrid substellar evolutionary models that account for cloud disappearance we infer the corresponding phase of over-luminosity may extend from a few hundred Myr up to a few Gyr and cause masses to to be over-estimated by up to 25%, even well after clouds disappear from view entirely. Thus, the range of of ages and spectral types affected by this potential systematic shift in luminosity evolution would encompass most known directly imaged gas-giants and field brown dwarfs.