We present a full derivation of the interaction Hamiltonian for holes in Silicon within the six-band envelope-function scheme, which appropriately describes the valence band close to the $boldsymbol{Gamma}$ point. The full structure of the single-hole eigenstates is taken into account, including the Bloch part, and the scattering processes caused by the Coulomb interaction are shown to be both intraband and interband; the interband terms are mostly short-ranged. In the asymptotic long-range limit, the effective potential tends to the screened Coulomb potential, and becomes purely intraband, as assumed in previous models. Our findings can be directly used for realistic exact-diagonalization calculations related to systems of interacting holes in Silicon nanostructures, such as quantum dots.