In this paper we study a $2+1$ dimensional system in which fermions are coupled to the self-dual topological vortex in $U(1) times U(1)$ Chern-Simons theory, where both $U(1)$ gauge symmetries are spontaneously broken. We consider two Abelian Higgs scalars with visible and hidden sectors coupled to a fermionic field through three interaction Lagrangians, where one of them violates the fermion number. Using a fine tuning procedure, we could obtain the number of the fermionic zero modes which is equal to the absolute value of the sum of the vortex numbers in the visible and hidden sectors.