The polarimetric observations on the protoplanetary disk around HL Tau have shown the scattering-induced polarization at ALMA Band 7, which indicates that the maximum dust size is $sim 100~{rm mu m}$, while the Spectral Energy Distribution (SED) has suggested that the maximum dust size is $sim$ mm. To solve the contradiction, we investigate the impact of differential settling of dust grains on the SED and polarization. If the disk is optically thick, longer observing wavelength traces more interior layer which would be dominated by larger grains. We find that, the SED of the center part of the HL Tau disk can be explained with mm-sized grains for a broad range of turbulence strength, while $160~{rm mu m}$-sized grains can explain barely only if the turbulence strength parameter $alpha_{rm t}$ is lower than $10^{-5}$. We also find that the observed polarization fraction can be potentially explained with the maximum dust size of $1~{rm mm}$ if $alpha_{rm t}lesssim10^{-5}$, although models with $160~{rm mu m}$-sized grains are also acceptable. However, if the maximum dust size is $sim3~{rm mm}$, the simulated polarization fraction is too low to explain the observations even if the turbulence strength is extremely small, indicating the maximum dust size of $lesssim1$ mm. The degeneracy between 100 ${rm mu m}$-sized and mm-sized grains can be solved by improving the ALMA calibration accuracy or polarimetric observations at (sub-)cm wavelengths.