An algebraic lower bound on the energy decay for solutions of the advection-diffusion equation in $mathbb{R}^d$ with $d=2,3$ is derived using the Fourier splitting method. Motivated by a conjecture on mixing of passive scalars in fluids, a lower bound on the $L^2-$ norm of the inverse gradient of the solution is obtained via gradient estimates and interpolation.