In this work, we investigate the representative case of the homometallic Co ludwigite Co$^{2+}_2$Co$^{3+}$O$_2$BO$_3$ ($Pbam$ space group) with four distinct Co crystallographic sites [$M1$-$M4$] surrounded by oxygen octahedra. The mixed-valent character of the Co ions up to at least $T=873$ K is verified through x-ray absorption near-edge structure (XANES) experiments. Single crystal x-ray diffraction (XRD) and neutron powder diffraction (NPD) confirm that the Co ions at the $M4$ site are much smaller than the others at low temperatures, consistent with a Co$^{3+}$ oxidation state at $M4$ and Co$^{2+}$ at the remaining sites. The size difference between the Co ions in the $M4$ and $M2$ sites is continuously reduced upon warming above $approx 370$ K, indicating a gradual charge redistribution within the $M4$-$M2$-$M4$ (424) ladder in the average structure. An increasing structural disorder, is noted above $approx 370$ K, The local Co-O distance distribution, revealed by Co $K$-edge Extended X-Ray Absorption Fine Structure (EXAFS) data and analyzed with an evolutionary algorithm method, is similar to that inferred from the XRD crystal structure below $approx 370$ K. At higher temperatures, the local Co-O distance distribution remains similar to that found at low temperatures, at variance with the average crystal structure obtained with XRD. We conclude that the oxidation states Co$^{2+}$ and Co$^{3+}$ are instantaneously well defined in a local atomic level at all temperatures, however the thermal energy promotes local defects in the charge-ordered configuration of the 424 ladders upon warming. These defects coalesce into a phase-segregated state within a narrow temperature interval ($475< T < 495$ K). Finally, a transition at $approx 500$ K revealed by differential scanning calorimetry (DSC) in the iron ludwigite Fe$_3$O$_2$BO$_3$ is discussed.