We present a study of the filamentary structure in the emission from the neutral atomic hydrogen (HI) at 21 cm across velocity channels in the 40 and 1.5-km/s resolution position-position-velocity cube resulting from the combination of the single-dish and interferometric observations in The HI/OH/Recombination (THOR) line survey. Using the Hessian matrix method in combination with tools from circular statistics, we find that the majority of the filamentary structures in the HI emission are aligned with the Galactic plane. Part of this trend can be assigned to long filamentary structures that are coherent across several velocity channels. However, we also find ranges of Galactic longitude and radial velocity where the HI filamentary structures are preferentially oriented perpendicular to the Galactic plane. These are located (i) around the tangent point of the Scutum spiral arm, $l approx 28^{circ}$ and $v_{rm LSR}approx 100$ km/s, (ii) toward $l approx 45^{circ}$ and $v_{rm LSR}approx 50$ km/s, (iii) around the Riegel-Crutcher cloud, and (iv) toward the terminal velocities. Comparison with numerical simulations indicates that the prevalence of horizontal filamentary structures is most likely the result of the large-scale dynamics and that vertical structures identified in (i) and (ii) may arise from the combined effect of supernova (SN) feedback and strong magnetic fields. The vertical filamentary structures in (iv) can be related to the presence of clouds from extra-planar HI gas falling back into the Galactic plane after being expelled by SNe. Our results indicate that a systematic characterization of the emission morphology toward the Galactic plane provides an unexplored link between the observations and the dynamical behaviour of the interstellar medium, from the effect of large-scale Galactic dynamics to the Galactic fountains driven by SNe.