We study the Fundamental Plane (FP) for a volume- and luminosity-limited sample of 560 early-type galaxies from the SAMI survey. Using r-band sizes and luminosities from new Multi-Gaussian Expansion (MGE) photometric measurements, and treating luminosity as the dependent variable, the FP has coefficients a=1.294$pm$0.039, b= 0.912$pm$0.025, and zero-point c= 7.067$pm$0.078. We leverage the high signal-to-noise of SAMI integral field spectroscopy, to determine how structural and stellar-population observables affect the scatter about the FP. The FP residuals correlate most strongly (8$sigma$ significance) with luminosity-weighted simple-stellar-population (SSP) age. In contrast, the structural observables surface mass density, rotation-to-dispersion ratio, Sersic index and projected shape all show little or no significant correlation. We connect the FP residuals to the empirical relation between age (or stellar mass-to-light ratio $Upsilon_star$) and surface mass density, the best predictor of SSP age amongst parameters based on FP observables. We show that the FP residuals (anti-)correlate with the residuals of the relation between surface density and $Upsilon_star$. This correlation implies that part of the FP scatter is due to the broad age and $Upsilon_star$ distribution at any given surface mass density. Using virial mass and $Upsilon_star$ we construct a simulated FP and compare it to the observed FP. We find that, while the empirical relations between observed stellar population relations and FP observables are responsible for most (75%) of the FP scatter, on their own they do not explain the observed tilt of the FP away from the virial plane.