Particle number violating interactions wash out the primordial asymmetry of particle number density generated by some interaction satisfying Sakharov conditions for baryogenesis. In this paper, we study how the primordial asymmetry evolves in time under the presence of particle number violating interactions and in the environment of expanding universe. We introduce a complex scalar model with particle number violating mass terms and calculate the time evolution of the particle number density with non-equilibrium quantum field theory. We show how the time evolution of the number density depends on parameters, including the chemical potential related with the particle number, temperature, size of the particle number violating mass terms, and the expansion rate of the universe. Depending upon whether the chemical potential is larger or smaller than the rest mass of the scalar particle, behaviors of the number density are very different to each other. When the chemical potential is smaller than the mass, the interference among the contribution of oscillators with various momenta reduces the number density in addition to the dilution due to the expansion of universe. In opposite case, the oscillation of the particle number density lasts for a long time and the cancellation due to the interference does not occur.