A finite group $P$ is said to be emph{primary} if $|P|=p^{a}$ for some prime $p$. We say a primary subgroup $P$ of a finite group $G$ satisfies the emph{Frobenius normalizer condition} in $G$ if $N_{G}(P)/C_{G}(P)$ is a $p$-group provided $P$ is $p$-group. In this paper, we determine the structure of a finite group $G$ in which every non-subnormal primary subgroup satisfies the Frobenius normalized condition. In particular, we prove that if every non-normal primary subgroup of $G$ satisfies the Frobenius condition, then $G/F(G)$ is cyclic and every maximal non-normal nilpotent subgroup $U$ of $G$ with $F(G)U=G$ is a Carter subgroup of $G$.