Model-independent constraints for the neutron-triton and proton-Helium-3 scattering lengths are calculated with a leading-order interaction derived from an effective field theory without explicit pions. Using the singlet neutron-proton scattering length, the deuteron, and the triton binding energy as input, the predictions $ants=9.2pm2.6 $fm, $antt=7.6pm1.6 $fm, $aphes=3.6pm0.32 $fm, and $aphet=3.1pm 0.23 $fm are obtained. The calculations employ the resonating group method and include the Coulomb interaction when appropriate. The theoretical uncertainty is assessed via a variation of the regulator parameter of the short-distance interaction from $400 $MeV to $1.6 $GeV. The phase-shift and scattering-length results for the proton-Helium-3 system are consistent with a recent phase shift analysis and with model calculations. For neutron-triton, the results for the scattering lengths in both singlet and triplet channels are significantly smaller than suggested by R-matrix and partial-wave-analysis extractions from data. For a better understanding of this discrepancy, the sensitivity of the low-energy four-body scattering system to variations in the neutron-neutron and proton-proton two-nucleon scattering lengths is calculated. Induced by strong charge-symmetry-breaking contact interactions, this dependence is found insignificant. In contrast, a strong correlation between the neutron-triton scattering length and the triton binding energy analogous to the Phillips line is found.