We present the rest-frame J- and H-band luminosity function (LF) of field galaxies, based on a deep multi-wavelength composite sample from the MUSYC, FIRES and FIREWORKS survey public catalogues, covering a total area of 450 arcmin^2. The availability of flux measurements in the Spitzer IRAC 3.6, 4.5, 5.8, and 8 um channels allows us to compute absolute magnitudes in the rest-frame J and H bands up to z=3.5 minimizing the dependence on the stellar evolution models. We compute the LF in the four redshift bins 1.5<z<2.0, 2.0<z<2.5, 2.5<z<3.0 and 3.0<z<3.5. Combining our results with those already available at lower redshifts, we find that (1) the faint end slope is consistent with being constant up to z=3.5, with alpha=-1.05+/-0.03 for the rest-frame J band and alpha=-1.15+/-0.02 for the rest-frame H band; (2) the normalization phi* decreases by a factor of 6 between z=0 and z~1.75 and by a factor 3 between z~1.75 and z=3.25; (3) the characteristic magnitude M* shows a brightening from z=0 to z~2 followed by a slow dimming to z=3.25. We finally compute the luminosity density (LD) in both rest-frame J and H bands. The analysis of our results together with those available in the literature shows that the LD is approximately constant up to z~1, and it then decreases by a factor of 6 up to z=3.5.