We report an investigation on structure and magnetic properties of the $S=3/2$ zigzag spin chain compound BaCoTe$_2$O$_7$. Neutron diffraction measurements reveal BaCoTe$_2$O$_7$ crystallizes in the noncentrosymmetric space group $Ama2$ with a canted $uparrowuparrowdownarrowdownarrow$ spin structure along the quasi-one-dimensional zigzag chain and a moment size of $1.89(2)mu_B$ at 2 K. Magnetic susceptibility and specific heat measurements yield an antiferromagnetic phase transition at $T_N=6.2$ K. A negative Curie-Weiss temperature $Theta_{CW}=-74.7(2)$ K and an empirical frustration parameter of $f=|Theta_text{CW}|/T_text{N}approx12$ is obtained from fitting the magnetic susceptibility, indicating antiferromagnetic interactions and strong magnetic frustration. By employing ultraviolet-visible absorption spectroscopy and first principles calculations, an indirect band gap of 2.68(2) eV is determined. We propose that the canted zigzag spin chain of BaCoTe$_2$O$_7$ may produce a change of the polarization via exchange striction mechanism.