With continuing improvements on the quality of fabricated quantum devices, it becomes increasingly crucial to analyze noisy quantum process in greater details such as characterizing the non-Markovianity in a quantitative manner. In this work, we propose an experimental protocol, termed Spectral Transfer Tensor Maps (SpecTTM), to accurately predict the RHP non-Markovian measure of any Pauli channels without state-preparation and measurement (SPAM) errors. In fact, for Pauli channels, SpecTTM even allows the reconstruction of highly-precised noise power spectrum for qubits. At last, we also discuss how SpecTTM can be useful to approximately characterize non-Markovianity of non-Pauli channels via Pauli twirling in an optimal basis.