The concept of fidelity has been introduced to characterize the stability of a quantum-mechanical system against perturbations. The fidelity amplitude is defined as the overlap integral of a wave packet with itself after the development forth and back under the influence of two slightly different Hamiltonians. It was shown by Prosen and Znidaric in the linear-response approximation that the decay of the fidelity is frozen if the Hamiltonian of the perturbation contains off-diagonal elements only. In the present work the results of Prosen and Znidaric are extended by a supersymmetry calculation to arbitrary strengths of the perturbation for the case of an unperturbed Hamiltonian taken from the Gaussian orthogonal ensemble and a purely unitary antisymmetric perturbation. It is found that for the exact calculation the freeze of fidelity is only slightly reduced as compared to the linear-response approximation. This may have important consequences for the design of quantum computers.