We study the anisotropic in-plane optical conductivity of detwinned Ba(Fe1-xCox)2As2 single crystals for x=0, 2.5% and 4.5% in a broad energy range (3 meV-5 eV) across their structural and magnetic transitions. For temperatures below the Neel transition, the topology of the reconstructed Fermi surface, combined with the distinct behavior of the scattering rates, determines the anisotropy of the low frequency optical response. For the itinerant charge carriers, we are able to disentangle the evolution of the Drude weights and scattering rates and to observe their enhancement along the orthorhombic antiferromagnetic a-axis with respect to the ferromagnetic b-axis. For temperatures above Ts, uniaxial stress leads to a finite in-plane anisotropy. The anisotropy of the optical conductivity, leading to a significant dichroism, extends to high frequencies in the mid- and near-infrared regions. The temperature dependence of the dichroism at all dopings scales with the anisotropy ratio of the dc conductivity, suggesting the electronic nature of the structural transition. Our findings bear testimony to a large nematic susceptibility that couples very effectively to the uniaxial lattice strain. In order to clarify the subtle interplay of magnetism and Fermi surface topology we compare our results with theoretical calculations obtained from density functional theory within the full-potential linear augmented plane-wave method.