The charge transfer compound (BEDT-TTF)$_2$Ag(CF$_3$)$_4$(TCE) crystallizes in three polymorphs with different alternating layers: While a phase with a $kappa$ packing motif has a low superconducting transition temperature of $T_c=2.6$ K, two phases with higher $T_c$ of $9.5$ and $11$ K are multi-layered structures consisting of $alpha$ and $kappa$ layers. We investigate these three systems within density functional theory and find that the $alpha$ layer shows different degrees of charge order for the two $kappa$-$alpha$ systems and directly influences the electronic behavior of the conducting $kappa$ layer. We discuss the origin of the distinct behavior of the three polymorphs and propose a minimal tight-binding Hamiltonian for the description of these systems based on projective molecular Wannier functions.