While the low-energy part of the hadronic light-by-light (HLbL) tensor can be constrained from data using dispersion relations, for a full evaluation of its contribution to the anomalous magnetic moment of the muon $(g-2)_mu$ also mixed- and high-energy regions need to be estimated. Both can be addressed within the operator product expansion (OPE), either for configurations where all photon virtualities become large or one of them remains finite. Imposing such short-distance constraints (SDCs) on the HLbL tensor is thus a major aspect of a model-independent approach towards HLbL scattering. Here, we focus on longitudinal SDCs, which concern the amplitudes containing the pseudoscalar-pole contributions from $pi^0$, $eta$, $eta$. Since these conditions cannot be fulfilled by a finite number of pseudoscalar poles, we consider a tower of excited pseudoscalars, constraining their masses and transition form factors from Regge theory, the OPE, and phenomenology. Implementing a matching of the resulting expressions for the HLbL tensor onto the perturbative QCD quark loop, we are able to further constrain our calculation and significantly reduce its model dependence. We find that especially for the $pi^0$ the corresponding increase of the HLbL contribution is much smaller than previous prescriptions in the literature would imply. Overall, we estimate that longitudinal SDCs increase the HLbL contribution by $Delta a_mu^text{LSDC}=13(6)times 10^{-11}$. This number does not include the contribution from the charm quark, for which we find $a_mu^{ctext{-quark}}=3(1)times 10^{-11}$.