Yb(Rh1-xCox)2Si2 is a model system to address two challenging problems in the field of strongly correlated electron systems: The first is the intriguing competition between ferromagnetic (FM) and antiferromagnetic (AFM) order when approaching a magnetic quantum critical point (QCP). The second is the occurrence of magnetic order along a very hard crystalline electric field (CEF) direction, i.e. along the one with the smallest available magnetic moment. Here, we present a detailed study of the evolution of the magnetic order in this system from a FM state with moments along the very hard c direction at x = 0.27 towards the yet unknown magnetic state at x = 0. We first observe a transition towards an AFM canted state with decreasing x and then to a pure AFM state. This confirms that the QCP in YbRh2Si2 is AFM, but the phase diagram is very similar to those observed in some inherently FM systems like NbFe2 and CeRuPO, which suggests that the basic underlying instability might be FM. Despite the huge CEF anisotropy the ordered moment retains a component along the c-axis also in the AFM state. The huge CEF anisotropy in Yb(Rh1-xCox)2Si2 excludes that this hard-axis ordering originates from a competing exchange anisotropy as often proposed for other heavy-fermion systems. Instead, it points to an order-by-disorder based mechanism.