The role of the magnetic field in the formation of the filamentary structures observed in the interstellar medium (ISM) is a debated topic. The Planck all-sky maps of linearly polarized emission from dust at 353GHz provide the required combination of imaging and statistics to study the correlation between the structures of the Galactic magnetic field and of interstellar matter, both in the diffuse ISM and in molecular clouds. The data reveal structures, or ridges, in the intensity map with counterparts in the Stokes Q and/or U maps. We focus on structures at intermediate and high Galactic latitudes with column density from $10^{20}$ to $10^{22}$ cm$^{-2}$. We measure the magnetic field orientation on the plane of the sky from the polarization data, and present an algorithm to estimate the orientation of the ridges from the dust intensity map. We use analytical models to account for projection effects. Comparing polarization angles on and off the structures, we estimate the mean ratio between the strengths of the turbulent and mean components of the magnetic field to be between 0.6 and 1.0, with a preferred value of 0.8. We find that the ridges are preferentially aligned with the magnetic field measured on the structures. This trend becomes more striking for increasing polarization fraction and decreasing column density. We interpret the increase of alignment with polarization fraction as a consequence of projections effects. The decrease of alignment for high column density is not due to a loss of correlation between the structures and the geometry of the magnetic field. In molecular complexes, we observe structures perpendicular to the magnetic field, which cannot be accounted for by projection effects. We discuss our results in the context of models and MHD simulations, which describe the formation of structures in the magnetized ISM.