We show a scenario for the cooling of compact stars considering the central source of Cassiopeia A (Cas A). The Cas A observation shows that the central source is a compact star with high effective temperature, and it is consistent with the cooling without exotic phases. The Cas A observation also gives the mass range of $M geq 1.5 M_odot$. It may conflict with the current cooling scenarios of compact stars that heavy stars show rapid cooling. We include the effect of the color superconducting (CSC) quark matter phase on the thermal evolution of compact stars. We assume the gap energy of CSC quark phase is large ($Delta gtrsim mathrm{10 MeV}$), and we simulate the cooling of compact stars. We present cooling curves obtained from the evolutionary calculations of compact stars: while heavier stars cool slowly, and lighter ones indicate the opposite tendency.