Background: The most significant source of background in direct dark matter searches are neutrons that scatter elastically from nuclei in the detectors sensitive volume. Experimental data for the elastic scattering cross section of neutrons from argon and neon, which are target materials of interest to the dark matter community, were previously unavailable. Purpose: Measure the differential cross section for elastic scattering of neutrons from argon and neon in the energy range relevant to backgrounds from (alpha,n) reactions in direct dark matter searches. Method: Cross-section data were taken at the Triangle Universities Nuclear Laboratory (TUNL) using the neutron time-of-flight technique. These data were fit using the spherical optical model. Results: The differential cross section for elastic scatting of neutrons from neon at 5.0 and 8.0 MeV and argon at 6.0 MeV was measured. Optical-model parameters for the elastic scattering reactions were determined from the best fit to these data. The total elastic scattering cross section for neon was found to differ by 6% at 5.0 MeV and 13% at 8.0 MeV from global optical-model predictions. Compared to a local optical-model for 40Ar, the elastic scattering cross section was found to differ from the data by 8% at 6.0 MeV. Conclusions: These new data are important for improving Monte-Carlo simulations and background estimates for direct dark matter searches and for benchmarking optical models of neutron elastic scattering from these nuclei.