The title compound Ba3RuTi2O9 crystallizes with a hexagonal unit cell. It contains layers of edge shared triangular network of Ru4+ (S=1) ions. Magnetic susceptibility chi(T) and heat capacity data show no long range magnetic ordering down to 1.8K. A Curie-Weiss (CW) fitting of chi(T) yields a large antiferromagnetic CW temperature theta_CW=-166K. However, in low field, a splitting of zero field cooled (ZFC) and field cooled (FC) chi(T) is observed below ~30K. Our measurements suggest that Ba3RuTi2O9 is a highly frustrated system but only a small fraction of the spins in this system undergo a transition to a frozen magnetic state below ~30K.