We characterize the algebraic structure of semi-direct product of cyclic groups, $Z_{N}rtimesZ_{p}$, where $p$ is an odd prime number which does not divide $q-1$ for any prime factor $q$ of $N$, and provide a polynomial-time quantum computational algorithm solving hidden symmetry subgroup problem of the groups.