Statistical physics cannot explain why a thermodynamic arrow of time exists, unless one postulates very special and unnatural initial conditions. Yet, we argue that statistical physics can explain why the thermodynamic arrow of time is universal, i.e., why the arrow points in the same direction everywhere. Namely, if two subsystems have opposite arrow-directions at a particular time, the interaction between them makes the configuration statistically unstable and causes a decay towards a system with a universal direction of the arrow of time. We present general qualitative arguments for that claim and support them by a detailed analysis of a toy model based on the bakers map.