Effects of the isospin-symmetry breaking (ISB) beyond mean-field Coulomb terms are systematically studied in nuclear masses near the $N=Z$ line. The Coulomb exchange contributions are calculated exactly. We use extended Skyrme energy density functionals (EDFs) with proton-neutron-mixed densities, to which we add new terms breaking the isospin symmetry. Two parameters associated with the new terms are determined by fitting mirror and triplet displacement energies (MDEs and TDEs) of isospin multiplets. The new EDFs reproduce MDEs for the $T=frac12$ doublets and $T=1$ triplets, and TDEs for the $T=1$ triplets. Relative strengths of the obtained isospin-symmetry-breaking terms {em are not} consistent with the differences in the $NN$ scattering lengths, $a_{nn}$, $a_{pp}$, and $a_{np}$. Based on low-energy experimental data, it seems thus impossible to delineate the strong-force ISB effects from beyond-mean-field Coulomb-energy corrections.