In this paper we provide performance guarantees for hypocoercive non-reversible MCMC samplers $X_t$ with invariant measure $mu^*$ and our results apply in particular to the Langevin equation, Hamiltonian Monte-Carlo, and the bouncy particle and zig-zag samplers. Specifically, we establish a concentration inequality of Bernstein type for ergodic averages $frac{1}{T} int_0^T f(X_t), dt$. As a consequence we provide performance guarantees: (a) explicit non-asymptotic confidence intervals for $int f dmu^*$ when using a finite time ergodic average with given initial condition $mu$ and (b) uncertainty quantification bounds, expressed in terms of relative entropy rate, on the bias of $int f dmu^*$ when using an alternative or approximate processes $widetilde{X}_t$. (Results in (b) generalize recent results (arXiv:1812.05174) from the authors for coercive dynamics.) The concentration inequality is proved by combining the approach via Feynmann-Kac semigroups first noted by Wu with the hypocoercive estimates of Dolbeault, Mouhot and Schmeiser (arXiv:1005.1495) developed for the Langevin equation and recently generalized to partially deterministic Markov processes by Andrieu et al. (arXiv:1808.08592)