Dimers and trimers of carbonyl sulfide (OCS) molecules embedded in helium nanodroplets are aligned by a linearly polarized 160 ps long moderately intense laser pulse and Coulomb exploded with an intense 40 fs long probe pulse in order to determine their structures. For the dimer, recording of 2D images of OCS$^+$ and S$^+$ ions and covariance analysis of the emission directions of the ions allow us to conclude that the structure is a slipped-parallel shape similar to the structure found for gas phase dimers. For the trimer, the OCS$^+$ ion images and corresponding covariance maps reveal the presence of a barrel-shaped structure (as in gas phase) but also other structures not present in the gas phase, most notably a linear chain structure.