Owing to the hybridization of ceriums localised 4$f$ electron and conduction band composed of $d$-electrons, cerium based intermetallics exhibit various kinds of magnetic interactions. In crystals, these can result in exotic types of magnetic ordering. In this study, we report a detailed single-crystal neutron diffraction study on CePdAl$_3$ and CePtAl$_3$. We have synthesized a large crystal of CePdAl$_3$, which crystallizes in a non-centrosymmetric, orthorhombic structure with space group $Cmc2_1$, a new, distorted variant of the tetragonal BaNiSn$_3$ structure observed in other Ce$T$Al$_3$ compounds, such as CePtAl$_3$. Low-temperature diffraction measurements showed that CePdAl$_3$ orders in a collinear antiferromagnetic structure below T$_N$=5.3 (1) K, with magnetic moments pointing along the $a$-axis direction and an ordered magnetic moment $mu$=1.64(3) $mu_B$/Ce$^{3+}$. Tetragonal CePtAl$_3$ shows a modulated, cycloidal type of ordering with $vec{k}=(frac{2}{3},0,0)$, and a transition temperature T$_N$=3.2 K. Symmetry analysis allows two types of ordering, which show modulation of both amplitude and direction of magnetic moments. These results allow to conclude that in Ce$T$Al$_3$ system the orthorhombic distortion ($T$=Pd, Ag) releases some underlying magnetic frustration that results in modulated types of magnetic ordering in tetragonal compounds ($T$=Cu,Au,Pt).