We study surface states of topological crystalline insulators and superconductors protected by inversion symmetry. These fall into the category of higher-order topological insulators and superconductors which possess surface states that propagate along one-dimensional curves (hinges) or are localized at some points (corners) on the surface. We show that the surface states of higher-order topological insulators and superconductors can be thought of as globally irremovable topological defects and provide a complete classification of these inversion-protected phases in any spatial dimension for the ten symmetry classes by means of a layer construction. Furthermore, we discuss possible physical realizations of such states starting with a time-reversal invariant topological insulator (class AII) in three dimensions or a time-reversal invariant topological superconductor (class DIII) in two or three dimensions. The former can be used to build a three-dimensional second-order topological insulator which exhibits one-dimensional chiral or helical modes propagating along opposite edges, whereas the latter enables the construction of three-dimensional third-order or two-dimensional second-order topological superconductors hosting Majorana zero modes localized to two opposite corners. Being protected by inversion, such states are not pinned to a specific pair of edges or corners thus offering the possibility of controlling their location by applying inversion-symmetric perturbations such as magnetic field.