We present a detailed investigation of the magnetic properties of complex vanadium phosphates M(VO)2(PO4)2 (M = Ca, Sr) by means of magnetization, specific heat, 31P NMR measurements, and band structure calculations. Experimental data evidence the presence of ferro- and antiferromagnetic interactions in M(VO)2(PO4)2 resulting in a nearly vanishing Curie-Weiss temperature theta_{CW} < 1 K that contrasts with the maximum of magnetic susceptibility at 3 K. Specific heat and NMR measurements also reveal weak exchange couplings with the thermodynamic energy scale J_c = 10-15 K. Additionally, the reduced maximum of the magnetic specific heat indicates strong frustration of the spin system. Band structure calculations show that the spin systems of the M(VO)2(PO4)2 compounds are essentially three-dimensional with the frustration caused by competing ferro- and antiferromagnetic interactions. Both calcium and strontium compounds undergo antiferromagnetic long-range ordering at T_N = 1.5 K and 1.9 K, respectively. The spin model reveals an unusual example of controllable frustration in three-dimensional magnetic systems.