As a potential window on transitions out of the ergodic, many-body-delocalized phase, we study the dephasing of weakly disordered, quasi-one-dimensional fermion systems due to a diffusive, non-Markovian noise bath. Such a bath is self-generated by the fermions, via inelastic scattering mediated by short-ranged interactions. We calculate the dephasing of weak localization perturbatively through second order in the bath coupling. However, the expansion breaks down at long times, and is not stabilized by including a mean-field decay rate, signaling a failure of the self-consistent Born approximation. We also consider a many-channel quantum wire where short-ranged, spin-exchange interactions coexist with screened Coulomb interactions. We calculate the dephasing rate, treating the short-ranged interactions perturbatively and the Coulomb interaction exactly. The latter provides a physical infrared regularization that stabilizes perturbation theory at long times, giving the first controlled calculation of quasi-1D dephasing due to diffusive noise. At first order in the diffusive bath coupling, we find an enhancement of the dephasing rate, but at second order we find a rephasing contribution. Our results differ qualitatively from those obtained via self-consistent calculations and are relevant in two different contexts. First, in the search for precursors to many-body localization in the ergodic phase. Second, our results provide a mechanism for the enhancement of dephasing at low temperatures in spin SU(2)-symmetric quantum wires, beyond the Altshuler-Aronov-Khmelnitsky result. The enhancement is possible due to the amplification of the triplet-channel interaction strength, and provides an additional mechanism that could contribute to the experimentally observed low-temperature saturation of the dephasing time.