We demonstrate that the sensitivity of high-precision pulsar timing experiments will be ultimately limited by the broadband intensity modulation that is intrinsic to the pulsars stochastic radio signal. That is, as the peak flux of the pulsar approaches that of the system equivalent flux density, neither greater antenna gain nor increased instrumental bandwidth will improve timing precision. These conclusions proceed from an analysis of the covariance matrix used to characterise residual pulse profile fluctuations following the template matching procedure for arrival time estimation. We perform such an analysis on 25 hours of high-precision timing observations of the closest and brightest millisecond pulsar, PSR J0437-4715. In these data, the standard deviation of the post-fit arrival time residuals is approximately four times greater than that predicted by considering the system equivalent flux density, mean pulsar flux and the effective width of the pulsed emission. We develop a technique based on principal component analysis to mitigate the effects of shape variations on arrival time estimation and demonstrate its validity using a number of illustrative simulations. When applied to our observations, the method reduces arrival time residual noise by approximately 20%. We conclude that, owing primarily to the intrinsic variability of the radio emission from PSR J0437-4715 at 20 cm, timing precision in this observing band better than 30 - 40 ns in one hour is highly unlikely, regardless of future improvements in antenna gain or instrumental bandwidth. We describe the intrinsic variability of the pulsar signal as stochastic wideband impulse modulated self-noise (SWIMS) and argue that SWIMS will likely limit the timing precision of every millisecond pulsar currently observed by Pulsar Timing Array projects as larger and more sensitive antennae are built in the coming decades.