Unitary $t$-designs are `good finite subsets of the unitary group $U(d)$ that approximate the whole unitary group $U(d)$ well. Unitary $t$-designs have been applied in randomized benchmarking, tomography, quantum cryptography and many other areas of quantum information science. If a unitary $t$-design itself is a group then it is called a unitary $t$-group. Although it is known that unitary $t$-designs in $U(d)$ exist for any $t$ and $d$, the unitary $t$-groups do not exist for $tgeq 4$ if $dgeq 3$, as it is shown by Guralnick-Tiep (2005) and Bannai-Navarro-Rizo-Tiep (BNRT, 2018). Explicit constructions of exact unitary $t$-designs in $U(d)$ are not easy in general. In particular, explicit constructions of unitary $4$-designs in $U(4)$ have been an open problem in quantum information theory. We prove that some exact unitary $(t+1)$-designs in the unitary group $U(d)$ are constructed from unitary $t$-groups in $U(d)$ that satisfy certain specific conditions. Based on this result, we specifically construct exact unitary $3$-designs in $U(3)$ from the unitary $2$-group $SL(3,2)$ in $U(3),$ and also unitary $4$-designs in $U(4)$ from the unitary $3$-group $Sp(4,3)$ in $U(4)$ numerically. We also discuss some related problems.