Lumping a Markov process introduces a coarser level of description that is useful in many contexts and applications. The dynamics on the coarse grained states is often approximated by its Markovian component. In this letter we derive finite-time bounds on the error in this approximation. These results hold for non-reversible dynamics and for probabilistic mappings between microscopic and coarse grained states.