Semiclassical Physics in gravitational scenario, in its first approximation (1st order) cares only for the expectation value of stress energy tensor and ignores the inherent quantum fluctuations thereof. In the approach of stochastic gravity, on the other hand, these matter fluctuations are supposed to work as the source of geometry fluctuations and have the potential to render the results from 1st order semiclassical physics irrelevant. We study the object of central significance in stochastic gravity, i.e. the noise kernel, for a wide class of Friedmann space-times. Through an equivalence of quantum fields on de Sitter space-time and those on generic Friedmann universes, we obtain the noise kernel through the correlators of Stress Energy Tensor (SET) for fixed co-moving but large physical distances. We show that in many Friedmann universes including the expanding universes, the initial quantum fluctuations, the universe is born with, may remain invariant and important even at late times. Further, we explore the cosmological space-times where even after long times the quantum fluctuations remain strong and become dominant over large physical distances, which the matter driven universe is an example of. The study is carried out in minimal as well as non-minimal interaction settings. Implications of such quantum fluctuations are discussed.