The recently discovered family of AV$_3$Sb$_5$ (A: K, Rb Cs) kagome metals possess a unique combination of nontrivial band topology, superconducting ground states, and signatures of electron correlations manifest via competing charge density wave order. Little is understood regarding the nature of the charge density wave (CDW) instability inherent to these compounds and the potential correlation with the accompanying onset of a large anomalous Hall response. To understand the impact of the CDW order on the electronic structure in these systems, we present quantum oscillation measurements on single crystals of CsV$_3$Sb$_5$. Our data provides direct evidence that the CDW invokes a substantial reconstruction of the Fermi surface pockets associated with the vanadium orbitals and the kagome lattice framework. In conjunction with density functional theory modeling, we are able to identify split oscillation frequencies originating from reconstructed pockets built from vanadium orbitals and Dirac-like bands. Complementary diffraction measurements are further able to demonstrate that the CDW instability has a correlated phasing between neighboring V$_3$Sb$_5$ planes. These results provide critical insights into the underlying CDW instability in AV$_3$Sb$_5$ kagome metals and support minimal models of CDW order arising from within the vanadium-based kagome lattice.