We establish a direct connection between inhomogeneous XX spin chains (or free fermion systems with nearest-neighbors hopping) and certain QES models on the line giving rise to a family of weakly orthogonal polynomials. We classify all such models and their associated XX chains, which include two families related to the Lame (finite gap) quantum potential on the line. For one of these chains, we numerically compute the Renyi bipartite entanglement entropy at half filling and derive an asymptotic approximation thereof by studying the models continuous limit, which turns out to describe a massless Dirac fermion on a suitably curved background. We show that the leading behavior of the entropy is that of a $c=1$ critical system, although there is a subleading $log(log N)$ correction (where $N$ is the number of sites) unusual in this type of models.