The antiferromagnet CaFe$_2$As$_2$ does not become superconducting when subject to ideal hydrostatic pressure conditions, where crystallographic and magnetic states also are well defined. By measuring electrical resistivity and magnetic susceptibilit
y under quasi-hydrostatic pressure, however, we find that a substantial volume fraction of the sample is superconducting in a narrow pressure range where collapsed tetragonal and orthorhombic structures coexist. At higher pressures, the collapsed tetragonal structure is stabilized, with the boundary between this structure and the phase of coexisting structures strongly dependent on pressure history. Fluctuations in magnetic degrees of freedom in the phase of coexisting structures appear to be important for superconductivity.
