Across many scientific and engineering disciplines, it is important to consider how much the output of a given system changes due to perturbations of the input. Here, we study the robustness of the ground states of $pm J$ spin glasses on random graphs to flips of the interactions. For a sparse graph, a dense graph, and the fully connected Sherrington-Kirkpatrick model, we find relatively large sets of interactions that generate the same ground state. These sets can themselves be analyzed as sub-graphs of the interaction domain, and we compute many of their topological properties. In particular, we find that the robustness of these sub-graphs is much higher than one would expect from a random model. Most notably, it scales in the same logarithmic way with the size of the sub-graph as has been found in genotype-phenotype maps for RNA secondary structure folding, protein quaternary structure, gene regulatory networks, as well as for models for genetic programming. The similarity between these disparate systems suggests that this scaling may have a more universal origin.