We present the exhaustive classification of surface states of topological insulators and superconductors protected by crystallographic magnetic point group symmetry in three spatial dimensions. Recently, Cornfeld and Chapman [Phys. Rev. B {bf 99}, 075105 (2019)] pointed out that the topological classification of mass terms of the Dirac Hamiltonian with point group symmetry is recast as the extension problem of the Clifford algebra, and we use their results extensively. Comparing two-types of Dirac Hamiltonians with and without the mass-hedgehog potential, we establish the irreducible character formula to read off which Hamiltonian in the whole $K$-group belongs to fourth-order topological phases, which are atomic insulators localized at the center of the point group.