Fix a semisimple Lie algebra g. Gaudin algebras are commutative algebras acting on tensor product multiplicity spaces for g-representations. These algebras depend on a parameter which is a point in the Deligne-Mumford moduli space of marked stable genus 0 curves. When the parameter is real, then the Gaudin algebra acts with simple spectrum on the tensor product multiplicity space and gives us a basis of eigenvectors. In this paper, we study the monodromy of these eigenvectors as the parameter varies within the real locus; this gives an action of the fundamental group of this moduli space, which is called the cactus group. We prove a conjecture of Etingof which states that the monodromy of eigenvectors for Gaudin algebras agrees with the action of the cactus group on tensor products of g-crystals. In fact, we prove that the coboundary category of normal g-crystals can be reconstructed using the coverings of the moduli spaces. Our main tool is the construction of a crystal structure on the set of eigenvectors for shift of argument algebras, another family of commutative algebras which act on any irreducible g-representation. We also prove that the monodromy of such eigenvectors is given by the internal cactus group action on g-crystals.