The Sun is replete with magnetic fields, with sunspots, pores and plage regions being their most prominent representatives on the solar surface. But even far away from these active regions, magnetic fields are ubiquitous. To a large extent, their importance for the thermodynamics in the solar photosphere is determined by the total magnetic flux. Whereas in low-flux quiet Sun regions, magnetic structures are shuffled around by the motion of granules, the high-flux areas like sunspots or pores effectively suppress convection, leading to a temperature decrease of up to 3000 K. The importance of magnetic fields to the conditions in higher atmospheric layers, the chromosphere and corona, is indisputable. Magnetic fields in both active and quiet regions are the main coupling agent between the outer layers of the solar atmosphere, and are therefore not only involved in the structuring of these layers, but also for the transport of energy from the solar surface through the corona to the interplanetary space. Consequently, inference of magnetic fields in the photosphere, and especially in the chromosphere, is crucial to deepen our understanding not only for solar phenomena such as chromospheric and coronal heating, flares or coronal mass ejections, but also for fundamental physical topics like dynamo theory or atomic physics. In this review, we present an overview of significant advances during the last decades in measurement techniques, analysis methods, and the availability of observatories, together with some selected results. We discuss the problems of determining magnetic fields at smallest spatial scales, connected with increasing demands on polarimetric sensitivity and temporal resolution, and highlight some promising future developments for their solution.