We develop a deep learning approach to extract ray directions at discrete locations by analyzing highly oscillatory wave fields. A deep neural network is trained on a set of local plane-wave fields to predict ray directions at discrete locations. The resulting deep neural network is then applied to a reduced-frequency Helmholtz solution to extract the directions, which are further incorporated into a ray-based interior-penalty discontinuous Galerkin (IPDG) method to solve the Helmholtz equations at higher frequencies. In this way, we observe no apparent pollution effects in the resulting Helmholtz solutions in inhomogeneous media. Our 2D and 3D numerical results show that the proposed scheme is very efficient and yields highly accurate solutions.