Non-Markovian effects are important in modeling the behavior of open quantum systems arising in solid-state physics, quantum optics as well as in study of biological and chemical systems. A common approach to the analysis of such systems is to approximate the non-Markovian environment by discrete bosonic modes thus mapping it to a Lindbladian or Hamiltonian simulation problem. While systematic constructions of such modes have been proposed in previous works [D. Tamascelli et al, PRL (2012), A. W. Chin et al, J. of Math. Phys (2010)], the resulting approximation lacks rigorous convergence guarantees. In this paper, we initiate a rigorous study of the convergence properties of these methods. We show that under some physically motivated assumptions on the system-environment interaction, the finite-time dynamics of the non-Markovian open quantum system computed with a sufficiently large number of modes is guaranteed to converge to the true result. Furthermore, we show that, for most physically interesting models of non-Markovian environments, the approximation error falls off polynomially with the number of modes. Our results lend rigor to numerical methods used for approximating non-Markovian quantum dynamics and allow for a quantitative assessment of classical as well as quantum algorithms in simulating non-Markovian quantum systems.