On 2004 Dec. 27, the soft gamma repeater (SGR) 1806-20 emitted the brightest giant flare (GF) ever detected from an SGR, with an (isotropic) energy release $sim 100$ times greater than the only two other known SGR GFs. It was followed by a very bright, fading radio afterglow. Extensive follow-up radio observations provided a wealth of information with unprecedented astrometric precision, revealing the temporal evolution of the source size, along with densely sampled light curves and spectra. Here we expand on our previous work on this source, by explaining these observations within one self-consistent dynamical model. In this scenario, the early radio emission is due to the outflow ejected during the GF energizing a thin shell surrounding a pre-existing cavity, where the observed steep temporal decay of the radio emission seen beginning on day 9 is attributed to the adiabatic cooling of the shocked shell. The shocked ejecta and external shell move outward together, driving a forward shock into the ambient medium, and are eventually decelerated by a reverse shock. As we show in Gelfand et al. (2005), the radio emission from the shocked external medium naturally peaks when significant deceleration occurs, and then decays relatively slowly. The dynamical modeling of the collision between the ejecta and the external shell together with the observed evolution of the source size (which is nicely reproduced in our model) suggest that most of the energy in the outflow was in mildly relativistic material, with an initial expansion velocity $v/c lesssim 0.7d_{15}$, for a distance of $15d_{15}$ kpc to SGR 1806-20. An initially highly relativistic outflow would not have produced a long coasting phase at a mildly relativistic expansion velocity, as was observed.