It has not been shown so far whether the diffuse Galactic polarized emission at frequencies relevant for cosmic microwave background (CMB) studies originates from nearby or more distant regions of our Galaxy. This questions previous attempts that have been made to constrain magnetic field models at local and large scales. The scope of this work is to investigate and quantify the contribution of the dusty and magnetized local interstellar medium to the observed emission that is polarized by thermal dust. We used stars as distance candles and probed the line-of-sight submillimeter polarization properties by comparing the emission that is polarized by thermal dust at submillimeter wavelengths and the optical polarization caused by starlight. We provide statistically robust evidence that at high Galactic latitudes ($|b| geq 60^circ$), the $353$ GHz polarized sky as observed by textit{Planck} is dominated by a close-by magnetized structure that extends between $200$ and $300$ pc and coincides with the shell of the Local Bubble. Our result will assist modeling the magnetic field of the Local Bubble and characterizing the CMB Galactic foregrounds.