Although experimental efforts have been active for about 30 years now, a direct laboratory observation of vacuum magnetic birefringence, an effect due to vacuum fluctuations, still needs confirmation. Indeed, the predicted birefringence of vacuum is $Delta n = 4.0times 10^{-24}$ @ 1~T. One of the key ingredients when designing a polarimeter capable of detecting such a small birefringence is a long optical path length within the magnetic field and a time dependent effect. To lengthen the optical path within the magnetic field a Fabry-Perot optical cavity is generally used with a finesse ranging from ${cal F} approx 10^4$ to ${cal F} approx7times 10^5$. Interestingly, there is a difficulty in reaching the predicted shot noise limit of such polarimeters. We have measured the ellipticity and rotation noises along with a Cotton-Mouton and a Faraday effect as a function of the finesse of the cavity of the PVLAS polarimeter. The observations are consistent with the idea that the cavity mirrors generate a birefringence-dominated noise whose ellipticity is amplified by the cavity itself. The optical path difference sensitivity at $10;$Hz is $S_{Delta{cal D}}=6times 10^{-19};$m$/sqrt{rm Hz}$, a value which we believe is consistent with an intrinsic thermal noise in the mirror coatings.