We present two independent calculations of the tight-binding parameters for a specific realization of the Haldane model with ultracold atoms. The tunneling coefficients up to next-to-nearest neighbors are computed ab-initio by using the maximally localized Wannier functions, and compared to analytical expressions written in terms of gauge invariant, measurable properties of the spectrum. The two approaches present a remarkable agreement and evidence the breakdown of the Peierls substitution: (i) the phase acquired by the next-to-nearest tunneling amplitude $t_{1}$ presents quantitative and qualitative differences with respect to that obtained by the integral of the vector field A, as considered in the Peierls substitution, even in the regime of low amplitudes of A; (ii) for larger values, also $|t_{1}|$ and the nearest-neighbor tunneling $t_{0}$ have a marked dependence on A. The origin of this behavior and its implications are discussed.