We have analyzed how much the $CP$ angles to be measured at B factories can deviate from the geometrical ones defined in unitarity triangle under the existence of new physics. The measurements are given in rephasing invariant form. If KM matrix is not a $3times 3$ and unitary matrix, $tildephi_1$ and $tildephi_3$ is affected, and the value of $tildephi_3$ depends on the decay mode. The deviation is constrained to be less than the experimental precision attained in the next decade by the available data of the magnitude of KM matrix elements. Deviation of the sum of three angles from $pi$ cannot be detected unless new physics contributes significantly to $b$ decay or $D$ meson system.