We propose a new method of pushing $Herschel$ to its faintest detection limits using universal trends in the redshift evolution of the far infrared over 24$mu$m colours in the well-sampled GOODS-North field. An extension to other fields with less mul
ti-wavelength information is presented. This method is applied here to raise the contribution of individually detected $Herschel$ sources to the cosmic infrared background (CIRB) by a factor 5 close to its peak at 250$mu$m and more than 3 in the 350$mu$m and 500$mu$m bands. We produce realistic mock $Herschel$ images of the deep PACS and SPIRE images of the GOODS-North field from the GOODS-$Herschel$ Key Program and use them to quantify the confusion noise at the position of individual sources, i.e., estimate a local confusion noise. Two methods are used to identify sources with reliable photometric accuracy extracted using 24$mu$m prior positions. The clean index (CI), previously defined but validated here with simulations, which measures the presence of bright 24$mu$m neighbours and the photometric accuracy index (PAI) directly extracted from the mock $Herschel$ images. After correction for completeness, thanks to our mock $Herschel$ images, individually detected sources make up as much as 54% and 60% of the CIRB in the PACS bands down to 1.1 mJy at 100$mu$m and 2.2 mJy at 160$mu$m and 55, 33, and 13% of the CIRB in the SPIRE bands down to 2.5, 5, and 9 mJy at 250$mu$m, 350$mu$m, and 500$mu$m, respectively. The latter depths improve the detection limits of $Herschel$ by factors of 5 at 250$mu$m, and 3 at 350$mu$m and 500$mu$m as compared to the standard confusion limit. Interestingly, the dominant contributors to the CIRB in all $Herschel$ bands appear to be distant siblings of the Milky Way ($z$$sim$0.96 for $lambda$$<$300$mu$m) with a stellar mass of $M_{star}$$sim$9$times$10$^{10}$M$_{odot}$.
