In this note, we consider special algebraic cycles on the Shimura variety S associated to a quadratic space V over a totally real field F, |F:Q|=d, of signature ((m,2)^{d_+},(m+2,0)^{d-d_+}), 1le d_+<d. For each n, 1le nle m, there are special cycles Z(T) in S, of codimension nd_+, indexed by totally positive semi-definite matrices with coefficients in the ring of integers O_F. The generating series for the classes of these cycles in the cohomology group H^{2nd_+}(S) are Hilbert-Siegel modular forms of parallel weight m/2+1. One can form analogous generating series for the classes of the special cycles in the Chow group CH^{nd_+}(S). For d_+=1 and n=1, the modularity of these series was proved by Yuan-Zhang-Zhang. In this note we prove the following: Assume the Bloch-Beilinson conjecture on the injectivity of Abel-Jacobi maps. Then the Chow group valued generating series for special cycles of codimension nd_+ on S is modular for all n with 1le nle m.