We present results for form factors of semileptonic decays of $D$ and $B$ mesons in 2+1 flavor lattice QCD using the MILC gauge configurations. With an improved staggered action for light quarks, we successfully reduce the systematic error from the chiral extrapolation. The results for $D$ decays are in agreement with experimental ones. The results for B decays are preliminary. Combining our results with experimental branching ratios, we then obtain the CKM matrix elements $|V_{cd}|$, $|V_{cs}|$, $|V_{cb}|$ and $|V_{ub}|$. We also check CKM unitarity, for the first time, using only lattice QCD as the theoretical input.