We present detailed systematic studies of structural transformations in thin liquid crystal films with the smectic-C to hexatic phase transition. For the first time all possible structures reported in the literature are observed for one material (5 O.6) at the variation of temperature and thickness. In unusual modulated structures the equilibrium period of stripes is twice with respect to the domain size. We interpret these patterns in the frame work of phenomenological Landau type theory, as equilibrium phenomena produced by a natural geometric frustration in a system having spontaneous splay distortion.