We present the K-band luminosity-halo mass relation, $L_{K,500}-M_{500,WL}$, for a subsample of 20 of the 100 brightest clusters in the XXL Survey observed with WIRCam at the Canada-France-Hawaii Telescope (CFHT). For the first time, we have measured this relation via weak-lensing analysis down to $M_{500,WL} =3.5 times 10^{13},M_odot$. This allows us to investigate whether the slope of the $L_K-M$ relation is different for groups and clusters, as seen in other works. The clusters in our sample span a wide range in mass, $M_{500,WL} =0.35-12.10 times 10^{14},M_odot$, at $0<z<0.6$. The K-band luminosity scales as $log_{10}(L_{K,500}/10^{12}L_odot) propto beta log_{10}(M_{500,WL}/10^{14}M_odot)$ with $beta = 0.85^{+0.35}_{-0.27}$ and an intrinsic scatter of $sigma_{lnL_K|M} =0.37^{+0.19}_{-0.17}$. Combining our sample with some clusters in the Local Cluster Substructure Survey (LoCuSS) present in the literature, we obtain a slope of $1.05^{+0.16}_{-0.14}$ and an intrinsic scatter of $0.14^{+0.09}_{-0.07}$. The flattening in the $L_K-M$ seen in previous works is not seen here and might be a result of a bias in the mass measurement due to assumptions on the dynamical state of the systems. We also study the richness-mass relation and find that group-sized halos have more galaxies per unit halo mass than massive clusters. However, the brightest cluster galaxy (BCG) in low-mass systems contributes a greater fraction to the total cluster light than BCGs do in massive clusters; the luminosity gap between the two brightest galaxies is more prominent for group-sized halos. This result is a natural outcome of the hierarchical growth of structures, where massive galaxies form and gain mass within low-mass groups and are ultimately accreted into more massive clusters to become either part of the BCG or one of the brighter galaxies. [Abridged]