We present a lattice QCD calculation of the axial $gamma W$-box diagrams relevant for the kaon semileptonic decays. We utilize a recently proposed method, which connects the electroweak radiative corrections in Sirlins representation to that in chiral perturbation theory. It allows us to use the axial $gamma W$-box correction in the SU(3) limit to obtain the low energy constants for chiral perturbation theory. From first principles our results confirm the previously used low energy constants provided by the minimal resonance model with a significant reduction in uncertainties.