Time-periodic (Floquet) drive is a powerful method to engineer quantum phases of matter, including fundamentally non-equilibrium states that are impossible in static Hamiltonian systems. One characteristic example is the anomalous Floquet insulator, which exhibits topologically quantized chiral edge states similar to a Chern insulator, yet is amenable to bulk localization. We study the response of this topological system to time-dependent noise, which breaks the topologically protecting Floquet symmetry. Surprisingly, we find that the quantized response, given by partially filling the fermionic system and measuring charge pumped per cycle, remains quantized up to finite noise amplitude. We trace this robust topology to an interplay between diffusion and Pauli blocking of edge state decay, which we expect should be robust against interactions. We determine the boundaries of the topological phase for a system with spatial disorder numerically through level statistics, and corroborate our results in the limit of vanishing disorder through an analytical Floquet superoperator approach. This approach suggests an interpretation of the state of the system as a non-Hermitian Floquet topological phase. We comment on quantization of other topological responses in the absence of Floquet symmetry and potential experimental realizations.