We introduce the notion of combinatorial gauge symmetry -- a local transformation that includes single spin rotations plus permutations of spins (or swaps of their quantum states) -- that preserve the commutation and anti-commutation relations among the spins. We show that Hamiltonians with simple two-body interactions contain this symmetry if the coupling matrix is a Hadamard matrix, with the combinatorial gauge symmetry being associated to the automorphism of these matrices with respect to monomial transformations. Armed with this symmetry, we address the physical problem of how to build quantum spin liquids with physically accessible interactions. In addition to its intrinsic physical significance, the problem is also tied to that of how to build topological qubits.