We examine critically how tightly the density dependence of nuclear symmetry energy esym is constrained by the universal equation of state (EOS) of the unitary Fermi gas $E_{rm{UG}}(rho)$ considering currently known uncertainties of higher order parameters describing the density dependence of the Equation of State of isospin-asymmetric nuclear matter. We found that $E_{rm{UG}}(rho)$ does provide a useful lower boundary for the esym. However, it does not tightly constrain the correlation between the magnitude $E_{rm{sym}}(rho_0)$ and slope $L$ unless the curvature $K_{rm{sym}}$ of the symmetry energy at saturation density $rho_0$ is more precisely known. The large uncertainty in the skewness parameters affects the $E_{rm{sym}}(rho_0)$ versus $L$ correlation by the same almost as significantly as the uncertainty in $K_{rm{sym}}$.