For $N=Z$ odd-odd nuclei, a three-body model assuming two valence particles and an inert core can provide an understanding of pairing correlations in the ground state and spin-isospin excitations. However, since residual core-nucleon interactions can have a significant impact on these quantities, the inclusion of core excitations in the model is essential for useful calculation to be performed. The effect of core excitations must be included in order to gain a detailed understanding of both the ground state and spin-isospin properties of these systems. To this end, we include the vibrational excitation of the core nucleus in our model. We solve the three-body core-nucleon-nucleon problem including core vibrational states to obtain the nuclear ground state as well as spin-isospin excitations. The spin-isospin excitations are examined from the point of view of SU(4) multiplets. By including the effect of core excitation, several experimental quantities of $N=Z$ odd-odd nuclei are better described, and the root mean square distances between proton and neutron and that between the center of mass of proton and neutron and core nucleus increase. Large $B$($M1$) and $B$(GT) observed for $^{18}$F and $^{40}$Ca were explained in terms of the SU(4) symmetry. The core nucleus is meaningfully broken by the residual core-nucleon interactions, and various quantities concerning spin-isospin excitations as well as the ground state become consistent with experimental data. Including the core excitation in the three-body model is thus important for a more detailed understanding of nuclear structure.