Garoufalidis and Levine defined a filtration for 3-manifolds equipped with some degree 1 map ($mathbb{Z}pi$-homology equivalence) to a fixed 3-manifold $N$ and showed that there is a natural surjection from a space of $pi=pi_1N$-decorated graphs to the graded quotient of the filtration over $mathbb{Z}[frac{1}{2}]$. In this paper, we show that in the case of $N=T^3$ the surjection of Garoufalidis--Levine is actually an isomorphism over $mathbb{Q}$. For the proof, we construct a perturbative invariant by applying Fukayas Morse homotopy theoretic construction to a local system of the quotient field of $mathbb{Q}pi$. The first invariant is an extension of the Casson invariant to $mathbb{Z}pi$-homology equivalences to the 3-torus. The results of this paper suggest that there is a highly nontrivial equivariant quantum invariants for 3-manifolds with $b_1=3$. We also discuss some generalizations of the perturbative invariant for other target spaces $N$.