We study four dimensional large-N SU(N) Yang-Mills theory coupled to adjoint overlap fermions on a single site lattice. Lattice simulations along with perturbation theory show that the bare quark mass has to be taken to zero as one takes the continuum limit in order to be in the physically relevant center-symmetric phase. But, it seems that it is possible to take the continuum limit with any renormalized quark mass and still be in the center-symmetric physics. We have also conducted a study of the correlations between Polyakov loop operators in different directions and obtained the range for the Wilson mass parameter that enters the overlap Dirac operator.