The single layered manganite Pr$_{0.22}$Sr$_{1.78}$MnO$_4$ undergoes structural transition from high temperature tetragonal phase to low temperature orthorhombic phase below room temperature. The orthorhombic phase was reported to have two structural variants with slightly different lattice parameters and Mn-3$d$ levels show orbital ordering within both the variants, albeit having mutually perpendicular ordering axis. In addition to orbital ordering, the orthorhombic variants also order antiferromagnetically with different Neel temperatures. Our magnetic investigation on the polycrystalline sample of Pr$_{0.22}$Sr$_{1.78}$MnO$_4$ shows large thermal hysteresis indicating the first order nature of the tetragonal to orthorhombic transition. We observe magnetic memory, large relaxation, frequency dependent ac susceptbility and aging effects at low temperature, which indicate spin glass like magnetic ground state in the sample. The glassy magnetic state presumably arises from the interfacial frustration of orthorhombic domains with orbital and spin orderings playing crucial role toward the competing magnetic interactions.