With decreasing temperature Sr$_2$VO$_4$ undergoes two structural phase transitions, tetragonal-to-orthorhombic-to-tetragonal, without long-range magnetic order. Recent experiments suggest, that only at very low temperature Sr$_{2}$VO$_{4}$ might enter some, yet unknown, phase with long-range magnetic order, but without orthorhombic distortion. By combining relativistic density functional theory with an extended spin-1/2 compass-Heisenberg model we find an antiferromagnetic single-stripe ground state with highly competing exchange interactions, involving a non negligible inter-layer coupling, which places the system at the crossover between between the XY and Heisenberg picture. Most strikingly, we find a strong two-site spin-compass exchange anisotropy which is relieved by the orthorhombic distortion induced by the spin stripe order. Based on these results we discuss the origin of the hidden order phase and the possible formation of a spin-liquid at low temperatures.