Ferromagnetic insulating La$_2$CoMnO$_{6-epsilon}$ (LCMO) epitaxial thin films grown on top of SrTiO$_3$ (001) substrates presents a strong magnetic anisotropy favoring the out of plane orientation of the magnetization with a strong anisotropy field ($sim 70$ kOe for film thickness of about 15 nm) and with a coercive field of about 10 kOe. The anisotropy can be tuned by modifying the oxygen content of the film which indirectly has two effects on the unit cell: i) change of the orientation of the LCMO crystallographic axis over the substrate (from c in-plane to c out-of-plane) and ii) shrinkage of the out of plane cell parameter, which implies increasing tensile strain of the films. In contrast, LCMO films grown on (LaAlO$_3$)$_{0.3}$(Sr$_2$AlTaO$_6$)$_{0.7}$ and LaAlO$_3$ substrates (with a larger out-of-plane lattice parameter and compressive stress) display in-plane magnetic anisotropy. Thus, we link the strong magnetic anisotropy observed in La$_2$CoMnO$_{6-epsilon}$ to the relation between in-plane and out-of-plane parameters and so to the film stress.