We study the influence of a chaotic environment in the evolution of an open quantum system. We show that there is an inverse relation between chaos and non-Markovianity. In particular, we remark on the deep relation of the short time non-Markovian behavior with the revivals of the average fidelity amplitude-a fundamental quantity used to measure sensitivity to perturbations and to identify quantum chaos. The long time behavior is established as a finite size effect which vanishes for large enough environments.