We investigate the phase structure of three-flavor QCD in the presence of finite quark chemical potential $mu/Tlesssim1.2$ by using the non-perturbatively $O(a)$ improved Wilson fermion action on lattices with a fixed temporal extent $N_{rm t}=6$ and varied spatial linear extents $N_{rm s}=8,10,12$. Especially, we focus on locating the critical end point that characterizes the phase structure, and extracting the curvature of the critical line on the $mu$-$m_{pi}$ plane. For Wilson-type fermions, the correspondence between bare parameters and physical parameters is indirect. Hence we present a strategy to transfer the bare parameter phase structure to the physical one, in order to obtain the curvature. Our conclusion is that the curvature is positive. This implies that, if one starts from a quark mass in the region of crossover at zero chemical potential, one would encounter a first-order phase transition when one raises the chemical potential.