This paper addresses the issue of the determination of the frictional stress distribution from the inversion of the measured surface displacement field for sliding interfaces between a glass lens and a rubber (poly(dimethylsiloxane)) substrate. Experimental results show that high lateral strains are achieved at the periphery of the sliding contacts. As a consequence, an accurate inversion of the displacement field requires that finite strains and non linear response of the rubber substrate are taken into account. For that purpose, a Finite Element (FE) inversion procedure is implemented where the measured displacement field is applied as a boundary condition at the upper surface of a meshed body representing the rubber substrate. Normal pressure is also determined by the same way, if non-diverging values are assumed at the contact edge. This procedure is applied to linearly sliding contacts as well as on twisting contacts.