A simple graph G=(V,E) is 3-rigid if its generic bar-joint frameworks in R3 are infinitesimally rigid. Block and hole graphs are derived from triangulated spheres by the removal of edges and the addition of minimally rigid subgraphs, known as blocks, in some of the resulting holes. Combinatorial characterisations of minimal $3$-rigidity are obtained for these graphs in the case of a single block and finitely many holes or a single hole and finitely many blocks. These results confirm a conjecture of Whiteley from 1988 and special cases of a stronger conjecture of Finbow-Singh and Whiteley from 2013.