Er:YSO crystal is promising candidate with great variety of its potential applications in quantum information processing and quantum communications ranging from optical/microwave quantum memories to circuit QED and microwave-to-optics frequency converters. Some of the above listed applications require ultra-low temperature environment, i.e. temperatures $Tlesssim0.1~$K. Most of the experiments with erbium doped crystals have been so far carried out at temperatures above 1.5 K. Therefore, only little information is known about Er:YSO coherence properties at millikelvins. Here, we investigate optical decoherence of $^{167}$Er:Y$_2$SiO$_5$ crystal by performing 2- and 3-pulse echo experiments at sub-Kelvin temperature range and at weak and moderate magnetic fields. We show that the deep freezing of the crystal results in an increase of optical coherence time by one order of magnitude below 1.5 Kelvin at the field of $sim$0.2 T. We further describe the detailed investigation of the decoherence mechanisms in this regime.