Enabled by rapidly developing quantum technologies, it is possible to network quantum systems at a much larger scale in the near future. To deal with non-Markovian dynamics that is prevalent in solid-state devices, we propose a general transfer function based framework for modeling linear quantum networks, in which signal flow graphs are applied to characterize the network topology by flow of quantum signals. We define a noncommutative ring $mathbb{D}$ and use its elements to construct Hamiltonians, transformations and transfer functions for both active and passive systems. The signal flow graph obtained for direct and indirect coherent quantum feedback systems clearly show the feedback loop via bidirectional signal flows. Importantly, the transfer function from input to output field is derived for non-Markovian quantum systems with colored inputs, from which the Markovian input-output relation can be easily obtained as a limiting case. Moreover, the transfer function possesses a symmetry structure that is analogous to the well-know scattering transformation in sd picture. Finally, we show that these transfer functions can be integrated to build complex feedback networks via interconnections, serial products and feedback, which may include either direct or indirect coherent feedback loops, and transfer functions between quantum signal nodes can be calculated by the Riegles matrix gain rule. The theory paves the way for modeling, analyzing and synthesizing non-Markovian linear quantum feedback networks in the frequency-domain.