In-plane anisotropic ground states are ubiquitous in correlated solids such as pnictides, cuprates and manganites. They can arise from doping Mott insulators and compete with phases such as superconductivity, however their origins are debated. Strong coupling between lattice, charge, orbital and spin degrees of freedom results in simultaneous ordering of multiple parameters, masking the mechanism that drives the transition. We demonstrate that the anisotropic orbital domains in a manganite can be oriented by the polarization of a pulsed THz light field. Through the application of the Hubbard model, we show that domain control can be achieved either through field assisted hopping of charges or a field-induced modification of bond angles. Both routes enhance the local Coulomb repulsions which drive domain reorientation and the dominant mechanism is dictated by the equilibrium Mn-O-Mn bond angle. Our results highlight the key role played by the Coulomb interaction in driving orbital order in manganites and demonstrate how THz can be utilized in new ways to understand and manipulate anisotropic phases in a broad range of correlated materials.