We consider an extension of Zee-Babu model to explain the smallness of neutrino masses. (1) We extend the lepton number symmetry of the original model to local $B-L$ symmetry. (2) We introduce three Dirac dark matter candidates with flavor-dependent $B-L$ charges. After the spontaneous breaking of $B-L$, a discrete symmetry $Z_6$ remains, which guarantees the stability of dark matter. Then the model can explain the 3.5 keV X-ray line signal with decaying dark matter. We also introduce a real scalar field which is singlet under both the SM and $U(1)_{B-L}$ and can explain the current relic abundance of the Dirac fermionic DMs. If the mixing with the SM Higgs boson is small, it does not contribute to DM direct detection. The main contribution to the scattering of DM off atomic nuclei comes from the exchange of $U(1)_{B-L}$ gauge boson, $Z$, and is suppressed below current experimental bound when $Z$ mass is heavy ($gtrsim 10$ TeV). If the singlet scalar mass is about 0.1--10 MeV, DM self-interaction can be large enough to solve small scale structure problems in simulations with the cold DM, such as, the core-vs-cusp problem and too-big-to-fail problem.