The $mu$-$tau$ exchange symmetry in the neutrino mass matrix and its breaking as a perturbation are discussed. The exact $mu$-$tau$ symmetry restricts the 2-3 and 1-3 neutrino mixing angles as $theta_{23} = pi/4$ and $theta_{13} = 0$ at a zeroth order level. We claim that the $mu$-$tau$ symmetry breaking prefers a large CP violation to realize the observed value of $theta_{13}$ and to keep $theta_{23}$ nearly maximal, though an artificial choice of the $mu$-$tau$ breaking can tune $theta_{23}$, irrespective of the CP phase. We exhibit several relations among the deviation of $theta_{23}$ from $pi/4$, $theta_{13}$ and Dirac CP phase $delta$, which are useful to test the $mu$-$tau$ breaking models in the near future experiments. We also propose a concrete model to break the $mu$-$tau$ exchange symmetry spontaneously and its breaking is mediated by the gauge interactions radiatively in the framework of the extended gauge model with $B-L$ and $L_mu - L_tau$ symmetries. As a result of the gauge mediated $mu$-$tau$ breaking in the neutrino mass matrix, the artificial choice is unlikely, and a large Dirac CP phase is preferable.