Random matrix models provide a phenomenological description of a vast variety of physical phenomena. Prominent examples include the eigenvalue statistics of quantum (chaotic) systems, which are conveniently characterized using the spectral form factor (SFF). Here, we calculate the SFF of unitary matrix ensembles of infinite order with the weight function satisfying the assumptions of Szegos limit theorem. We then consider a parameter-dependent critical ensemble which has intermediate statistics characteristic of ergodic-to-nonergodic transitions such as the Anderson localization transition. This same ensemble is the matrix model of $U(N)$ Chern-Simons theory on $S^3$, and the SFF of this ensemble is proportional to the HOMFLY invariant of $(2n,2)$-torus links with one component in the fundamental and one in the antifundamental representation. This is one of a large class of ensembles arising from topological field and string theories which exhibit intermediate statistics. Indeed, the absence of a local order parameter suggests that it is natural to characterize ergodic-to-nonergodic transitions using topological tools, such as we have done here.