We performed mapping observations of the Class I protostellar binary system L1551 NE in the C$^{18}$O ($J$=3-2), $^{13}$CO ($J$=3-2), CS ($J$=7-6), and SO ($J_N$=7$_8$-6$_7$) lines with Atacama Submillimeter Telescope Experiment (ASTE). The ASTE C$^{18}$O data are combined with our previous SMA C$^{18}$O data, which show a $r sim$300-AU scale Keplerian disk around the protostellar binary system. The C$^{18}$O maps show a $sim$20000-AU scale protostellar envelope surrounding the central Keplerian circumbinary disk. The envelope exhibits a northeast (blue) - southwest (red) velocity gradient along the minor axis, which can be interpreted as a dispersing gas motion with an outward velocity of 0.3 km s$^{-1}$, while no rotational motion in the envelope is seen. In addition to the envelope, two $lesssim$4000 AU scale, high-velocity ($gtrsim$1.3 km s$^{-1}$) redshifted $^{13}$CO and CS emission components are found to $sim$40$^{primeprime}$ southwest and $sim$20$^{primeprime}$ west of the protostellar binary. These redshifted components are most likely outflow components driven from the neighboring protostellar source L1551 IRS 5, and are colliding with the envelope in L1551 NE. The net momentum, kinetic and internal energies of the L1551 IRS 5 outflow components are comparable to those of the L1551 NE envelope, and the interactions between the outflows and the envelope are likely to cause the dissipation of the envelope and thus suppression of the further growth of the mass and mass ratio of the central protostellar binary in L1551 NE.