Graphitic nitrogen-doped graphene is an excellent platform to study scattering processes of massless Dirac fermions by charged impurities, in which high mobility can be preserved due to the absence of lattice defects through direct substitution of carbon atoms in the graphene lattice by nitrogen atoms. In this work, we report on electrical and magnetotransport measurements of high-quality graphitic nitrogen-doped graphene. We show that the substitutional nitrogen dopants in graphene introduce atomically sharp scatters for electrons but long-range Coulomb scatters for holes and, thus, graphitic nitrogen-doped graphene exhibits clear electron-hole asymmetry in transport properties. Dominant scattering processes of charge carriers in graphitic nitrogen-doped graphene are analyzed. It is shown that the electron-hole asymmetry originates from a distinct difference in intervalley scattering of electrons and holes. We have also carried out the magnetotransport measurements of graphitic nitrogen-doped graphene at different temperatures and the temperature dependences of intervalley scattering, intravalley scattering and phase coherent scattering rates are extracted and discussed. Our results provide an evidence for the electron-hole asymmetry in the intervalley scattering induced by substitutional nitrogen dopants in graphene and shine a light on versatile and potential applications of graphitic nitrogen-doped graphene in electronic and valleytronic devices.