Pulsars show two classes of rotational irregularities that can be used to understand neutron-star interiors and magnetospheres: glitches and timing noise. Here we present an analysis of the Vela pulsar spanning nearly 21 yr of observation and including 8 glitches. We identify the relative pulse number of all of the observations between glitches, with the only pulse-number ambiguities existing over glitch events. We use the phase coherence of the timing solution to simultaneously model the timing noise and glitches in a Bayesian framework, allowing us to select preferred models for both. We find the glitches can be described using only permanent and transient changes in spin frequency, i.e., no step changes in frequency derivative. For all of the glitches, we only need two exponentially decaying changes in spin frequency to model the transient components. In contrast to previous studies, we find that the dominant transient components decay on a common $approx$ 1300 d time scale, and that a larger fraction ( $gtrsim 25%$) of glitch amplitudes are associated with these transient components. We also detect shorter-duration transient components of $approx$ 25 d, as previously observed, but are limited in sensitivity to events with shorter durations by the cadence of our observations. The timing noise is well described by a steep power-law process that is independent of the glitches and subdominant to the glitch recovery. The braking index is constrained to be $<$ 8 with 95% confidence. This methodology can be used to robustly measure the properties of glitches and timing noise in other pulsars.