In molecular-based quantum-spin-liquid candidate EtMe$_3$Sb[Pd(dmit)$_2$]$_2$ with two-dimensional $S$=1/2 triangular lattice, a finite residual linear term in the thermal conductivity, $kappa_0/Tequivkappa/T (T rightarrow 0)$, has been observed and attributed to the presence of itinerant gapless excitations. Here we show that the data of $kappa$ measured in several single crystals are divided into two groups with and without the residual linear term. In the first group with finite $kappa_0/T$, the phonon thermal conductivity $kappa_{ph}$ is comparable to that of other organic compounds. In these crystals, the phonon mean free path $ell_{ph}$ saturates at low temperatures, being limited by sample size. On the other hand, in the second group with zero $kappa_0/T$, $kappa_{ph}$ is one order of magnitude smaller than that in the first group, comparable to that of amorphous solids. In contrast to the first group, $ell_{ph}$ shows a glassy-like non-saturating behavior at low temperatures. These results suggest that the crystals with long $ell_{ph}$ are required to discuss the magnetic excitations by thermal conductivity measurements.