Physicists become acquainted with special functions early in their studies. Consider our perennial model, the harmonic oscillator, for which we need Hermite functions, or the Laguerre functions in quantum mechanics. Here we choose a particular number theoretical function, the Riemann zeta function and examine its influence in the realm of physics and also how physics may be suggestive for the resolution of one of mathematics most famous unconfirmed conjectures, the Riemann Hypothesis. Does physics hold an essential key to the solution for this more than hundred-year-old problem? In this work we examine numerous models from different branches of physics, from classical mechanics to statistical physics, where this function plays an integral role. We also see how this function is related to quantum chaos and how its pole-structure encodes when particles can undergo Bose-Einstein condensation at low temperature. Throughout these examinations we highlight how physics can perhaps shed light on the Riemann Hypothesis. Naturally, our aim could not be to be comprehensive, rather we focus on the major models and aim to give an informed starting point for the interested Reader.