We discuss in a critical way the physical foundations of geometric structure of relativistic theories of gravity by the so-called Ehlers-Pirani-Schild formalism. This approach provides a natural interpretation of the observables showing how relate them to General Relativity and to a large class of Extended Theories of Gravity. In particular we show that, in such a formalism, geodesic and causal structures of space-time can be safely disentangled allowing a correct analysis in view of observations and experiment. As specific case, we take into account the case of f(R) gravity.