Measurements of the phase of two-photon matrix elements are presented for near-resonant two-color ionization of helium. A tunable, narrow-bandwidth, near-infrared (NIR) laser source is used for extreme ultra-violet (XUV) high-harmonic generation (HHG). The 15th harmonic of the laser is used within (1+1) XUV+NIR two-photon ionization, and tuned in and out of resonance with members of the 1s$n$p $^1$P$_1$ ($n=3,4,5$) Rydberg series. Rapid changes in the phase of the two-photon matrix elements around resonances and at the mid-way point between two resonances are observed, encoding the relative importance of resonant and near-resonant two-color ionization. Similar effects are observed for (1+2) XUV+NIR three-photon ionization. The experimental results are compared to a perturbative model and numerical solution of the time-dependent Schrodinger equation (TDSE) in the single active electron (SAE) approximation.