We report the measurement of the photoionization cross sections of the 5S${}_{1/2}$ and 5P${}_{3/2}$ states of ${}^{87}$Rb in a two-species Hg and Rb magneto-optical trap (MOT) by the cooling laser for Hg. The photoionization cross sections of Rb in the 5S${}_{1/2}$ and 5P${}_{3/2}$ states at 253.7~nm are determined to be $1^{+4.3}_{-1}times10^{-20}~text{cm}^2$ and $4.63(30)times 10^{-18}text{cm}^2$, respectively. To measure the 5S${}_{1/2}$ and 5P${}_{3/2}$ states fractions in the MOT we detected photoionization rate of the 5P${}_{3/2}$ state by an additional 401.5~nm laser. The photoionization cross section of Rb in the 5P${}_{3/2}$ state at 401.5~nm is determined to be $text{1.18(10)}times10^{-17}~text{cm}^2$.