Hybrid organic-inorganic perovskites have emerged as very promising materials for photonic applications, thanks to the great synthetic versatility that allows to tune their optical properties. In the two-dimensional (2D) crystalline form, these materials behave as multiple quantum-well heterostructures with stable excitonic resonances up to room temperature. In this work strong light-matter coupling in 2D perovskite single-crystal flakes is observed, and the polarization-dependent exciton-polariton response is used to disclose new excitonic features. For the first time, an out-of-plane component of the excitons is observed, unexpected for such 2D systems and completely absent in other layered materials, such as transition-metal dichalcogenides. By comparing different hybrid perovskites with the same inorganic layer but different organic interlayers, it is shown how the nature of the organic ligands controllably affects the out-of-plane exciton-photon coupling. Such vertical dipole coupling is particularly sought in those systems, e.g. plasmonic nanocavities, in which the direction of the field is usually orthogonal to the material sheet. Organic interlayers are shown to affect also the strong birefringence associated to the layered structure, which is exploited in this work to completely rotate the linear polarization degree in only few microns of propagation, akin to what happens in metamaterials.