Reversible topotactic transitions between oxygen-vacancy-ordered structures in transition metal oxides provide a promising strategy for active manipulation of material properties. While transformations between various oxygen-deficient phases have been attained in bulk ABO$_{3-delta}$ perovskites, substrate clamping restricts the formation of distinct ordering patterns in epitaxial films. Using in-situ scanning transmission electron microscopy (STEM), we image a thermally driven reversible transition in La$_{0.5}$Sr$_{0.5}$CoO$_{3-delta}$ films on SrTiO$_3$ from a multidomain brownmillerite (BM) structure to a uniform phase wherein oxygen vacancies order in every third CoO$_x$ plane. Because temperature cycling is performed over a limited temperature range (25 {deg}C - 385 {deg}C), the oxygen deficiency parameter $delta$ does not vary measurably. Under constant $delta$, the topotactic transition proceeds via local reordering of oxygen vacancies driven by thermal strain. Atomic-resolution imaging reveals a two-step process whereby alternating vertically and horizontally oriented BM domains first scale in size to accommodate the strain induced by different thermal expansions of La$_{0.5}$Sr$_{0.5}$CoO$_{3-delta}$ and SrTiO$_3$, before the new phase nucleates and quickly grows above 360 {deg}C. Upon cooling, the film transform back to the mixed BM phase. As the structural transition is fully reversible and $delta$ does not change upon temperature cycling, we rule out electron-beam irradiation during STEM as the driving mechanism. Instead, our findings demonstrate that thermal strain can solely drive topotactic phase transitions in perovskite oxide films, presenting opportunities for switchable ionic devices.