We have studied polycrystalline Yb4LiGe4, a ternary variant of the R5T4 family of layered compounds characterized by a very strong coupling between the magnetic and crystallographic degrees of freedom. The system is mixed valent, with non-magnetic Yb2+ and magnetic Yb3+ present, and is characterized by coexisting ferromagnetic and antiferromagnetic correlations. We present measurements of resistivity, AC-susceptibility, specific heat, and muon spin relaxation (muSR), below 1 K. The low temperature measurements suggest a transition to a mesoscopically inhomogeneous magnetically ordered state below 2 K characterized by fluctuations well below the ordering temperature. This unusual state is believed to result from the enhanced two-dimensionality produced by Li substitution and frustration effects inherent in the Yb sub-lattice geometry.