We present a novel 3D topological insulator, termed the Takagi topological insulator (TTI), which is protected by the sublattice symmetry and spacetime inversion symmetry. The symmetries enable the Takagi factorization in the Hamiltonian space. Due t
o the intrinsic O(N) gauge symmetry in the Takagi factorization, a Z2 topological invariant is formulated. We examine the physical consequences of the topological invariant through a Dirac model, which exhibits exotic bulk boundary correspondence. The most stable phases are a number of novel third-order topological insulators featured with odd inversion pairs of corners hosting zero-modes. Furthermore, the nontrivial bulk invariant corresponds to a rich cross-boundary-order phase diagram with a hierarchical cellular structure. Each cell with its own dimensionality corresponds to a certain configuration of boundary states, which could be of mixed orders.
For conventional topological phases, the boundary gapless modes are determined by bulk topological invariants. Based on developing an analytic method to solve higher-order boundary modes, we present $PT$-invariant $2$D topological insulators and $3$D
topological semimetals that go beyond this bulk-boundary correspondence framework. With unchanged bulk topological invariant, their first-order boundaries undergo transitions separating different phases with second-order-boundary zero-modes. For the $2$D topological insulator, the helical edge modes appear at the transition point for two second-order topological insulator phases with diagonal and off-diagonal corner zero-modes, respectively. Accordingly, for the $3$D topological semimetal, the criticality corresponds to surface helical Fermi arcs of a Dirac semimetal phase. Interestingly, we find that the $3$D system generically belongs to a novel second-order nodal-line semimetal phase, possessing gapped surfaces but a pair of diagonal or off-diagonal hinge Fermi arcs.
