We study the $P-V$ criticality and phase transition in the extended phase space of charged Gauss-Bonnet black holes in anti-de Sitter space, where the cosmological constant appears as a dynamical pressure of the system and its conjugate quantity is the thermodynamic volume of the black hole. The black holes can have a Ricci flat ($k=0$), spherical ($k=1$), or hyperbolic ($k=-1$) horizon. We find that for the Ricci flat and hyperbolic Gauss-Bonnet black holes, no $P-V$ criticality and phase transition appear, while for the black holes with a spherical horizon, even when the charge of the black hole is absent, the $P-V$ criticality and the small black hole/large black hole phase transition will appear, but it happens only in $d=5$ dimensions; when the charge does not vanish, the $P-V$ criticality and the small black hole/large phase transition always appear in $d=5$ dimensions; in the case of $dge 6$, to have the $P-V$ criticality and the small black hole/large black hole phase transition, there exists an upper bound for the parameter $b=widetilde{alpha}|Q|^{-2/(d-3)}$, where $tilde {alpha}$ is the Gauss-Bonnet coefficient and $Q$ is the charge of the black hole. We calculate the critical exponents at the critical point and find that for all cases, they are the same as those in the van der Waals liquid-gas system.