A unitary $t$-design is a powerful tool in quantum information science and fundamental physics. Despite its usefulness, only approximate implementations were known for general $t$. In this paper, we provide for the first time quantum circuits that generate exact unitary $t$-designs for any $t$ on an arbitrary number of qubits. Our construction is inductive and is of practical use in small systems. We then introduce a $t$-th order generalization of randomized benchmarking ($t$-RB) as an application of exact $2t$-designs. We particularly study the $2$-RB in detail and show that it reveals self-adjointness of quantum noise, a new metric related to the feasibility of quantum error correction (QEC). We numerically demonstrate that the $2$-RB in one- and two-qubit systems is feasible, and experimentally characterize background noise of a superconducting qubit by the $2$-RB. It is shown from the experiment that interactions with adjacent qubits induce the noise that may result in an obstacle toward the realization of QEC.