The Milnor Problem (modified) in the theory of group growth asks whether any finite presented group of vanishing algebraic entropy has at most polynomial growth. We show that a positive answer to the Milnor Problem (modified) is equivalent to the Nilpotency Conjecture in Riemannian geometry: given $n, d>0$, there exists a constant $epsilon(n,d)>0$ such that if a compact Riemannian $n$-manifold $M$ satisfies that Ricci curvature $op{Ric}_Mge -(n-1)$, diameter $dge op{diam}(M)$ and volume entropy $h(M)<epsilon(n,d)$, then the fundamental group $pi_1(M)$ is virtually nilpotent. We will verify the Nilpotency Conjecture in some cases, and we will verify the vanishing gap phenomena for more cases i.e., if $h(M)<epsilon(n,d)$, then $h(M)=0$.