In the present paper, we prove that any finite non-trivial irreducible module over a rank two Lie conformal algebra $mathcal{H}$ is of rank one. We also describe the actions of $mathcal{H}$ on its finite irreducible modules explicitly. Moreover, we show that all finite non-trivial irreducible modules of finite Lie conformal algebras whose semisimple quotient is the Virasoro Lie conformal algebra are of rank one.