We argue that the topological charge density wave phase in the quasi-2D Kagome superconductor AV$_3$Sb$_5$ is a chiral flux phase. Considering the symmetry of the Kagome lattice, we show that the chiral flux phase has the lowest energy among those states which exhibit $2times2$ charge orders observed experimentally. This state breaks the time-reversal symmetry and displays anomalous Hall effect. The explicit pattern of the density of this state in real space is calculated. These results are supported by recent experiments and suggest that these materials are a new platform to investigate the interplay between topology, superconductivity and electron-electron correlations.