The Casimir effect is a general phenomenon in physics, which arises when the vacuum fluctuation of an arbitrary field is modified by static or slowly varying boundary. However, its spin version is rarely addressed, mainly due to the fact that a macro
scopic boundary in quantum spin systems is hard to define. In this article, we explore the spin Casimir effect induced by the zero-point fluctuation of spin waves in a general non-collinear ordered quantum antiferromagnet. This spin Casimir effect results in a spin torque between local spins and further causes various singular and divergent results in the framework of spin-wave theory, which invalidate the standard $1/S$ expansion procedure. To avoid this dilemma, we develop a self-consistent spin-wave expansion approach, which preserves the spin-wave expansion away from singularities and divergence. A detailed spin-wave analysis of the antiferromagnetic spin-1/2 Heisenberg model on a spatially anisotropic triangular lattice is undertaken within our approach. Our results indicate that the spiral order is only stable in the region $0.5<alpha<1.2$, where $alpha$ is the ratio of the coupling constants. In addition, the instability in the region $1.2<alpha<2$ is owing to the spin Casimir effect instead of the vanishing sublattice magnetization. And this extended spiral instable region may host some quantum disordered phases besides the quantum order by disorder induced Neel phase. Furthermore, our method provides an efficient and convenient tool that can estimate the correct exchange parameters and outline the quantum phase diagrams, which can be useful for experimental fitting processes in frustrated quantum magnets.
