We report the evolution of nematic fluctuations in FeSe$_{1-x}$S$_x$ single crystals as a function of Sulfur content $x$ across the nematic quantum critical point (QCP) $x_csim$ 0.17 via Raman scattering. The Raman spectra in the $B_{1g}$ nematic channel consist of two components, but only the low energy one displays clear fingerprints of critical behavior and is attributed to itinerant carriers. Curie-Weiss analysis of the associated nematic susceptibility indicates a substantial effect of nemato-elastic coupling which shifts the location of the nematic QCP. We argue that this lattice-induced shift likely explains the absence of any enhancement of the superconducting transition temperature at the QCP. The presence of two components in the nematic fluctuations spectrum is attributed to the dual aspect of electronic degrees of freedom in Hunds metals, with both itinerant carriers and local moments contributing to the nematic susceptibility.