In this work we investigate the late-time stationary states of open quantum systems coupled to a thermal reservoir in the strong coupling regime. In general such systems do not necessarily relax to a Boltzmann distribution if the coupling to the thermal reservoir is non-vanishing or equivalently if the relaxation timescales are finite. Using a variety of non-equilibrium formalisms valid for non-Markovian processes, we show that starting from a product state of the closed system = system + environment, with the environment in its thermal state, the open system which results from coarse graining the environment will evolve towards an equilibrium state at late-times. This state can be expressed as the reduced state of the closed system thermal state at the temperature of the environment. For a linear (harmonic) system and environment, which is exactly solvable, we are able to show in a rigorous way that all multi-time correlations of the open system evolve towards those of the closed system thermal state. Multi-time correlations are especially relevant in the non-Markovian regime, since they cannot be generated by the dynamics of the single-time correlations. For more general systems, which cannot be exactly solved, we are able to provide a general proof that all single-time correlations of the open system evolve to those of the closed system thermal state, to first order in the relaxation rates. For the special case of a zero-temperature reservoir, we are able to explicitly construct the reduced closed system thermal state in terms of the environmental correlations.