Let $G$ be a finite, undirected $d$-regular graph and $A(G)$ its normalized adjacency matrix, with eigenvalues $1 = lambda_1(A)geq dots ge lambda_n ge -1$. It is a classical fact that $lambda_n = -1$ if and only if $G$ is bipartite. Our main result provides a quantitative separation of $lambda_n$ from $-1$ in the case of Cayley graphs, in terms of their expansion. Denoting $h_{out}$ by the (outer boundary) vertex expansion of $G$, we show that if $G$ is a non-bipartite Cayley graph (constructed using a group and a symmetric generating set of size $d$) then $lambda_n ge -1 + ch_{out}^2/d^2,,$ for $c$ an absolute constant. We exhibit graphs for which this result is tight up to a factor depending on $d$. This improves upon a recent result by Biswas and Saha who showed $lambda_n ge -1 + h_{out}^4/(2^9d^8),.$ We also note that such a result could not be true for general non-bipartite graphs.