Quantum key distribution (QKD) which enables the secure distribution of symmetric keys between two legitimate parties is of great importance in future network security. Access network that connects multiple end-users with one network backbone can be combined with QKD to build security for end-users in a scalable and cost-effective way. Though previous QKD access networks are all implemented in the upstream direction, in this paper, we prove that downstream access network can also be constructed by using continuous-variable (CV) QKD. The security of the CV-QKD downstream access network is analyzed in detail, where we show the security analysis is secure against other parties in the network. The security analysis we proved corresponds to the downstream access network where only passive beamsplitter is sufficient to distribute the quantum signals and no other active controls are demanded. Moreover, standard CV-QKD systems can be directly fitted in the downstream access network, which makes it more applicable for practical implementations. Numerous simulation results are provided to demonstrate the performance of the CV-QKD downstream access network, where up to 64 end-users are shown to be feasible to access the network. Our work provides the security analysis framework for realizing QKD in the downstream access network which will boost the diversity for constructing practical QKD networks.