The binding energy differences of the valence proton and neutron of the mirror nuclei, $^{15}$O -- $^{15}$N, $^{17}$F -- $^{17}$O, $^{39}$Ca -- $^{39}$K and $^{41}$Sc -- $^{41}$Ca, are calculated using the quark-meson coupling (QMC) model. The calculation involves nuclear structure and shell effects explicitly. It is shown that binding energy differences of a few hundred keV arise from the strong interaction, even after subtracting all electromagnetic corrections. The origin of these differences may be ascribed to the charge symmetry breaking effects set in the strong interaction through the u and d current quark mass difference.