The Weyl semimetal NbP was found to exhibit topological Fermi arcs and exotic magneto-transport properties. Here, we report on magnetic quantum-oscillation measurements on NbP and construct the 3D Fermi surface with the help of band-structure calculations. We reveal a pair of spin-orbit-split electron pockets at the Fermi energy and a similar pair of hole pockets, all of which are strongly anisotropic. The Fermi surface well explains the linear magnetoresistance observed in high magnetic fields by the quantum-limit scenario. The Weyl points that are located in the $k_z approx pi/c$ plane are found to exist 5 meV above the Fermi energy. Therefore, we predict that the chiral anomaly effect can be realized in NbP by electron doping to drive the Fermi energy to the Weyl points.