Barlowite Cu$_4$(OH)$_6$FBr shows three-dimensional (3D) long-range antiferromagnetism, which is fully suppressed in Cu$_3$Zn(OH)$_6$FBr with a kagome quantum spin liquid ground state. Here we report systematic studies on the evolution of magnetism in the Cu$_{4-x}$Zn$_x$(OH)$_{6}$FBr system as a function of $x$ to bridge the two limits of Cu$_4$(OH)$_6$FBr ($x$=0) and Cu$_3$Zn(OH)$_6$FBr ($x$=1). Neutron-diffraction measurements reveal a hexagonal-to-orthorhombic structural change with decreasing temperature in the $x$ = 0 sample. While confirming the 3D antiferromagnetic nature of low-temperature magnetism, the magnetic moments on some Cu$^{2+}$ sites on the kagome planes are found to be vanishingly small, suggesting strong frustration already exists in barlowite. Substitution of interlayer Cu$^{2+}$ with Zn$^{2+}$ with gradually increasing $x$ completely suppresses the bulk magnetic order at around $x$ = 0.4, but leaves a local secondary magnetic order up to $xsim 0.8$ with a slight decrease in its transition temperature. The high-temperature magnetic susceptibility and specific heat measurements further suggest that the intrinsic magnetic properties of kagome spin liquid planes may already appear from $x>0.3$ samples. Our results reveal that the Cu$_{4-x}$Zn$_x$(OH)$_6$FBr may be the long-thought experimental playground for the systematic investigations of the quantum phase transition from a long-range antiferromagnet to a topologically ordered quantum spin liquid.