We have studied Josephson junctions with barriers prepared from the Heusler compound Cu$_2$MnAl. In the as-prepared state the Cu$_2$MnAl layers are non ferromagnetic and the critical Josephson current density $j_{c}$ decreases exponentially with the thickness of the Heusler layers $d_{F}$. On annealing the junctions at 240degree C the Heusler layers develop ferromagnetic order and we observe a dependence $j_{c}(d_{F}$) with $j_{c}$ strongly enhanced and weakly thickness dependent in the thickness range 7.0 nm < $d_{F}$ < 10.6 nm. We attribute this feature to a triplet component in the superconducting pairing function generated by the specific magnetization profile inside thin Cu$_2$MnAl layers.