CaV$_2$O$_4$ is a spin-1 antiferromagnet, where the magnetic vanadium ions are arranged on quasi-one-dimensional (1D) zig-zag chains with potentially frustrated antiferromagnetic exchange interactions. High temperature susceptibility and single-crystal neutron diffraction measurements are used to deduce the non-collinear magnetic structure, dominant exchange interactions and orbital configurations. The results suggest that at high temperatures CaV$_2$O$_4$ behaves as a Haldane chain, but at low temperatures, orbital ordering lifts the frustration and it becomes a spin-1 ladder.