Recently, the bi-layer Kagome lattice material Ca$_{10}$Cr$_7$O$_{28}$ has been shown to be a quasi-two-dimensional quantum spin liquid (QSL) where the frustration arises from a balance between competing ferromagnetic and antiferromagnetic exchange within a bi-layer. In an attempt to understand what happens when this balance is disturbed, we present a magnetic dilution study. Specifically, we have synthesized Ca$_{10}$(Cr$_{1-x}$V$_x$)$_7$O$_{28}$ (0 $leq$ x $leq$ 0.5) where magnetic Cr$^{5+}$ ($S = 1/2$) is partially replaced by non-magnetic V$^{5+}$ ($S = 0$). We also synthesized the fully non-magnetic isostructural material Ca$_{10}$V$_7$O$_{27.5}$. We report a detailed structural, magnetic and heat capacity study on these materials. A monotonic increase in the unit cell parameters is found for the Ca$_{10}$(Cr$_{1-x}$V$_x$)$_7$O$_{28}$ materials with increasing $x$. An order of magnitude decrease in the Curie-Weiss temperature from $4$ to $0.5$~ K is found for the partial V substituted samples, which indicates a relative increase in antiferromagnetic exchange with increase in V content. However, despite this change in the relative balance in the exchange interactions and the large disorder introduced, no magnetic ordering or spin-glass state is observed down to $2$~K in the V substituted samples. The QSL state of the parent compound thus seems surprisingly robust against these large perturbations.