We discuss an inverse seesaw model based on right-handed fermion specific $U(1)$ gauge symmetry and $A_4$-modular symmetry. These symmetries forbid unnecessary terms and restrict structures of Yukawa interactions which are relevant to inverse seesaw mechanism. Then we can obtain some predictions in neutrino sector such as Dirac-CP phase and sum of neutrino mass, which are shown by our numerical analysis. Besides the relation among masses of heavy pseudo-Dirac neutrino can be obtained since it is also restricted by the modular symmetry. We also discuss implications to lepton flavor violation and collider physics in our model.