Plasma-based electron and positron wakefield acceleration has made great strides in the past decade. However one major challenge for its applications to coherent light sources and colliders is the relatively large energy spread of the accelerated beams, currently at a few percent level. This energy spread is usually correlated with particle position in the beam arising from the longitudinal chirp of the wakefield amplitude. Therefore a dechirper is highly desirable for reducing this spread down to $sim0.1%$ level, while at the same time for maintaining the emittance of the accelerated beam. Here we propose that a low-density hollow channel plasma can act as a near-ideal dechirper for both electrons and positrons. We demonstrate the concept through large-scale three-dimensional particle-in-cell simulations. We show that the initial positive correlated energy spread (chirp) on the beam exiting a plasma accelerator can be compensated by the nearly linear self-wake induced by the beam in the hollow channel from few percent level down to $leq 0.1%$. Meanwhile, the beam emittance can be preserved due to the negligible transverse field inside the channel. This passive method may significantly improve the beam quality of plasma-based accelerators, paving the way for their applications to future compact free electron lasers and colliders.