Vertical magnetic fields have been known to exist in the internetwork region for decades, while the properties of horizontal magnetic fields have recently been extensively investigated with textit{Hinode}. Vertical and horizontal magnetic fields in the internetwork region are considered to be separate entities and have thus far not been investigated in a unified way. We discover clear positional association between the vertical and horizontal magnetic fields in the internetwork region with textit{Hinode}. Essentially all of the horizontal magnetic patches are associated with the vertical magnetic patches. Alternatively, half of the vertical magnetic patches accommodate the horizontal magnetic patches. These horizontal patches are located around the borders of the vertical patches. The intrinsic magnetic field strength as obtained with the Stokes $V$ line ratio inside the horizontal patches is weak, and is in sub-equipartition field regime ($B<700$ G), while the field strength outside the horizontal patches ranges from weak to strong (kG) fields. Vertical magnetic patches are known to be concentrated on mesogranular and supergranular boundaries, while the horizontal magnetic patches are found only on the mesogranular boundaries. These observations provide us with new information on the origin of the vertical and horizontal internetwork magnetic fields, in a unified way. We conjecture that internetwork magnetic fields are provided by emergence of small-scale flux tubes with bipolar footpoints, and the vertical magnetic fields of the footpoints are intensified to kG fields due to convective collapse. Resultant strong vertical fields are advected by the supergranular flow, and eventually form the network fields.