We use a statistical sample of galaxy clusters from a large cosmological $N$-body$+$hydrodynamics simulation to examine the relation between morphology, or shape, of the X-ray emitting intracluster medium (ICM) and the mass accretion history of the galaxy clusters. We find that the mass accretion rate (MAR) of a cluster is correlated with the ellipticity of the ICM. The correlation is largely driven by material accreted in the last $sim 4.5$~Gyr, indicating a characteristic time-scale for relaxation of cluster gas. Furthermore, we find that the ellipticity of the outer regions ($Rsim R_{rm 500c}$) of the ICM is correlated with the overall MAR of clusters, while ellipticity of the inner regions ($lesssim 0.5 R_{rm 500c}$) is sensitive to recent major mergers with mass ratios of $geq 1:3$. Finally, we examine the impact of variations in cluster mass accretion history on the X-ray observable-mass scaling relations. We show that there is a {it continuous/} anti-correlation between the residuals in the $T_x-M$ relation and cluster MARs, within which merging and relaxed clusters occupy extremes of the distribution rather than form two peaks in a bi-modal distribution, as was often assumed previously. Our results indicate the systematic uncertainties in the X-ray observable-mass relations can be mitigated by using the information encoded in the apparent ICM ellipticity.