We investigate gravitational waves with sub-nanoHz frequencies ($10^{-11}$ Hz $lesssim f_{rm GW} lesssim 10^{-9}$ Hz) from the spatial distribution of the spin-down rates of milli-second pulsars. As we suggested in Yonemaru et al. 2018, gravitational waves from a single source induces the bias in the observed spin-down rates of pulsars depending on the relative direction between the source and pulsar. To improve the constraints on the time derivative of gravitational-wave amplitude obtained in our previous work (Kumamoto et al. 2019), we adopt a more sophisticated statistical method called the Mann-Whitney U test. Applying our method to the ATNF pulsar catalogue, we first found that the current data set is consistent with no GW signal from any direction in the sky. Then, we estimate the effective angular resolution of our method to be $(66~{rm deg})^2$ by studying the probability distribution of the test statistic. Finally, we investigate gravitational-wave signal from the Galactic Centre and M87 and, comparing simulated mock data sets with the real pulsar data, we obtain the upper bounds on the time derivative as $dot{h}_{rm GC} < 8.9 times 10^{-19} {rm s}^{-1}$ for the Galactic Centre and $dot{h}_{rm M87} < 3.3 times 10^{-19} {rm s}^{-1}$ for M87, which are stronger than the ones obtained in Kumamoto et al. 2019 by factors of 7 and 25, respectively.