The origin of both the Ising chain magnetism and ferroelectricity in Ca$_3$CoMnO$_6$ is studied by $ab$ $initio$ electronic structure calculations and x-ray absorption spectroscopy. We find that Ca$_3$CoMnO$_6$ has the alternate trigonal prismatic Co$^{2+}$ and octahedral Mn$^{4+}$ sites in the spin chain. Both the Co$^{2+}$ and Mn$^{4+}$ are in the high spin state. In addition, the Co$^{2+}$ has a huge orbital moment of 1.7 $mu_B$ which is responsible for the significant Ising magnetism. The centrosymmetric crystal structure known so far is calculated to be unstable with respect to exchange striction in the experimentally observed $uparrowuparrowdownarrowdownarrow$ antiferromagnetic structure for the Ising chain. The calculated inequivalence of the Co-Mn distances accounts for the ferroelectricity.