We present a comprehensive study on the low-lying states of neutron-rich Er, Yb, Hf, and W isotopes across the $N=126$ shell with a multi-reference covariant density functional theory. Beyond mean-field effects from shape mixing and symmetry restoration on the observables that are relevant for understanding quadrupole collectivity and underlying shell structure are investigated. The general features of low-lying states in closed-shell nuclei are retained in these four isotopes around $N=126$, even though the shell gap is overall quenched by about 30% with the beyond mean-field effects. These effects are consistent with the previous generator-coordinate calculations based on Gogny forces, but much smaller than that predicted by the collective Hamiltonian calculation. It implies that the beyond mean-field effects on the $r$-process abundances before the third peak at $Asim195$ might be more moderate than that found in A. Arcones and G. F. Bertsch, Phys. Rev. Lett. 108, 151101 (2012).