Self-interacting dark matter (SIDM) has gathered growing attention as a solution to the small scale problems of the collisionless cold dark matter (DM). We investigate the SIDM using stellar kinematics of 23 ultra-faint dwarf (UFD) galaxies with the phenomenological SIDM halo model. The UFDs are DM-dominated and have less active star formation history. Accordingly, they are the ideal objects to test the SIDM, as their halo profiles are least affected by the baryonic feedback processes. We found no UFDs favor non-zero self-interaction and some provide stringent constraints within the simple SIDM modeling. Our result challenges the simple modeling of the SIDM, which urges further investigation of the subhalo dynamical evolution of the SIDM.