For all types of N=4 anti-de Sitter (AdS) supersymmetry in three dimensions, we construct manifestly supersymmetric actions for Abelian vector multiplets and explain how to extend the construction to the non-Abelian case. Manifestly N=4 supersymmetric Yang-Mills (SYM) actions are explicitly given in the cases of (2,2) and critical (4,0) AdS supersymmetries. The N=4 vector multiplets and the corresponding actions are then reduced to (2,0) AdS superspace, in which only N=2 supersymmetry is manifest. Using the off-shell structure of the N=4 vector multiplets, we provide complete N=4 SYM actions in (2,0) AdS superspace for all types of N=4 AdS supersymmetry. In the case of (4,0) AdS supersymmetry, which admits a Euclidean counterpart, the resulting N=2 action contains a Chern-Simons term proportional to q/r, where r is the radius of AdS_3 and q is the R-charge of a chiral scalar superfield. The R-charge is a linear inhomogeneous function of X, an expectation value of the N=4 Cotton superfield. Thus our results explain the mysterious structure of N=4 supersymmetric Yang-Mills theories on S^3 discovered in arXiv:1401.7952. In the case of (3,1) AdS supersymmetry, which has no Euclidean counterpart, the SYM action contains both a Chern-Simons term and a chiral mass-like term. In the case of (2,2) AdS supersymmetry, which admits a Euclidean counterpart, the SYM action has no Chern-Simons and chiral mass-like terms.