The tomographic Alcock-Paczynski (AP) test is a robust large-scale structure (LSS) measurement that receives little contamination from the redshift space distortion (RSD). It has placed tight cosmological constraints by using small and intermediate clustering scales of the LSS data. However, previous works have neglected the cross-correlation among different redshift bins, which could cause the statistical uncertainty being underestimated by $sim$20%. In this work, we further improve this method by including this multi-redshifts full correlation. We apply it to the SDSS DR12 galaxies sample and find out that, for $Lambda$CDM, the combination of AP with the Planck+BAO dataset slightly reduces (within 1-$sigma$) $Omega_m$ to $0.304pm0.007$ (68.3% CL). This then leads to a larger $H_0$ and also mildly affects $Omega_b h^2$, $n_s$ and the derived parameters $z_*$, $r_*$, $z_{re}$ but not $tau$, $A_s$ and $sigma_8$. For the flat $w$CDM model, our measurement gives $Omega_m=0.301pm 0.010$ and $w=-1.090pm 0.047$, where the additional AP measurement reduces the error budget by $sim 25%$. When including more parameters into the analysis, the AP method also improves the constraints on $Omega_k$, $sum m_mu$, $N_{rm eff}$ by $20-30%$. Early universe parameters such as $dn_s/d{rm ln}k$ and $r$, however, are unaffected. Assuming the dark energy equation of state $w=w_0+w_a frac{z}{1+z}$, the Planck+BAO+SNIa+$H_0$+AP datasets prefer a dynamical dark energy at $approx1.5 sigma$ CL. Finally, we forecast the cosmological constraints expected from the DESI galaxy survey and find that combining AP with CMB+BAO method would improve the $w_0$-$w_a$ constraint by a factor of $sim 10$.