We prove that for any two continuous minimal (topologically free) actions of the infinite dihedral group on an infinite compact Hausdorff space, they are continuously orbit equivalent only if they are conjugate. We also show the above fails if we replace the infinite dihedral group with certain other virtually cyclic groups, e.g. the direct product of the integer group with any non-abelian finite simple group.