We perform a $z$-band survey for an optical counterpart of a binary neutron star coalescence GW170817 with Subaru/Hyper Suprime-Cam. Our untargeted transient search covers $23.6$ deg$^2$ corresponding to the $56.6%$ credible region of GW170817 and reaches the $50%$ completeness magnitude of $20.6$ mag on average. As a result, we find 60 candidates of extragalactic transients, including J-GEM17btc (a.k.a. SSS17a/DLT17ck). While J-GEM17btc is associated with NGC 4993 that is firmly located inside the 3D skymap of GW170817, the other 59 candidates do not have distance information in the GLADE v2 catalog or NASA/IPAC Extragalactic Database (NED). Among 59 candidates, 58 are located at the center of extended objects in the Pan-STARRS1 catalog, while one candidate has an offset. We present location, $z$-band apparent magnitude, and time variability of the candidates and evaluate the probabilities that they are located inside of the 3D skymap of GW170817. The probability for J-GEM17btc is $64%$ being much higher than those for the other 59 candidates ($9.3times10^{-3}-2.1times10^{-1}%$). Furthermore, the possibility, that at least one of the other 59 candidates is located within the 3D skymap, is only $3.2%$. Therefore, we conclude that J-GEM17btc is the most-likely and distinguished candidate as the optical counterpart of GW170817.