The inversion and volume effects on magnetism in a spinel-type magnetically frustrated compound, CoAl2O4, and its gallium-substituted system, CoAl2-xGaxO4, were investigated. Magnetically frustrated Co2+ with spin S = 3/2 on the tetrahedral site formed a diamond lattice in CoAl2O4 located in the vicinity of the magnetic phase boundary between Neel and spin-spiral states. In the Ga-substituted system, the number of Co ions, the so-called inversion h dominating the octahedral site, increased with increasing x. From comprehensive crystallographic, magnetic, and thermal measurements, increments of both volume and inversion strongly reduced the Neel point, while the latter also induced a spin-glass state above the critical value of hc = 0.09. In the spin glass state, h > hc, the orbital degree of freedom of Co2+ ions in the octahedral site appeared in the magnetic entropy, which couples strongly with that of spin, even above the magnetic transitions. Above h ~ hc, the field-induced quenched magnetic moment appeared above the transitions. Therefore, a short range ordered state emerged among the paramagnetic, antiferromganetic, and spin-glass states in the magnetic phase diagram.