In $e$-beam evaporated amorphous silicon ($a$-Si), the densities of two-level systems (TLS), $n_{0}$ and $overline{P}$, determined from specific heat $C$ and internal friction $Q^{-1}$ measurements, respectively, have been shown to vary by over three orders of magnitude. Here we show that $n_{0}$ and $overline{P}$ are proportional to each other with a constant of proportionality that is consistent with the measurement time dependence proposed by Black and Halperin and does not require the introduction of additional anomalous TLS. However, $n_{0}$ and $overline{P}$ depend strongly on the atomic density of the film ($n_{rm Si}$) which depends on both film thickness and growth temperature suggesting that the $a$-Si structure is heterogeneous with nanovoids or other lower density regions forming in a dense amorphous network. A review of literature data shows that this atomic density dependence is not unique to $a$-Si. These findings suggest that TLS are not intrinsic to an amorphous network but require a heterogeneous structure to form.