Among many quantum key distribution (QKD) protocols, the round-robin differential phase shift (RRDPS) protocol is unique in that it can upper-bound the amount of the information leakage without monitoring the signal disturbance. To expedite implementation of the protocol, however, the number of pulses forming a single block should be kept small, which significantly decreases the key rates in the original security proof. In the present paper, we refine the security proof of the RRDPS protocol in the finite-sized regime and achieve a tighter estimation for the information leakage without changing the original experimental setups. As a consequence, we obtain better key rates in both asymptotic and finite-sized cases while keeping the preferable features of the protocol, such as omission of phase randomization.