We report synthesis and single crystal measurements of magnetic, transport and thermal properties of single crystalline BaCo$_2$As$_2$ as well as first principles calculations of the electronic structure and magnetic behavior. These results show that BaCo$_2$As$_2$ is a highly renormalized paramagnet in proximity to a quantum critical point, presumably of ferromagnetic character and that BaFeNiAs$_2$ behaves similarly. These results are discussed in relation to the properties of Ba(Fe,Co)$_2$As$_2$ and Ba(Fe,Ni)$_2$As$_2$, which are superconducting for low Co and Ni concentrations.