$Out(F_n):=Aut(F_n)/Inn(F_n)$ denotes the outer automorphism group of the rank $n$ free group $F_n$. An element $phi$ of $Out(F_n)$ is polynomially growing if the word lengths of conjugacy classes in $F_n$ grow at most polynomially under iteration by $phi$. We restrict attention to the subset $UPG(F_n)$ of $Out(F_n)$ consisting of polynomially growing elements whose action on $H_1(F_n, Z)$ is unipotent. In particular, if $phi$ is polynomially growing and acts trivially on $H_1(F_n,Z_3)$ then $phi$ is in $UPG(F_n)$ and also every polynomially growing element of $Out(F_n)$ has a positive power that is in $UPG(F_n)$. In this paper we solve the conjugacy problem for $UPG(F_n)$. Specifically we construct an algorithm that takes as input $phi, psiin UPG(F_n)$ and outputs YES or NO depending on whether or not there is $thetain Out(F_n)$ such that $psi=thetaphitheta^{-1}$. Further, if YES then such a $theta$ is produced.