We present the redshift evolution of the restframe galaxy luminosity function (LF) in the red r, i, and z bands as derived from the FORS Deep Field (FDF). Using the deep and homogeneous I-band selected dataset of the FDF we are able to follow the red LFs over the redshift range 0.5 < z < 3.5. The results are based on photometric redshifts for 5558 galaxies derived from the photometry in 9 filters achieving an accuracy of Delta z / (z_spec+1) ~ 0.03 with only ~ 1 % outliers. Because of the depth of the FDF we can give relatively tight constraints on the faint-end slope alpha of the LF: The faint-end of the red LFs does not show a large redshift evolution and is compatible within 1 sigma to 2 sigma with a constant slope over the redshift range 0.5 < z < 2.0. Moreover, the slopes in r, i, and z are very similar with a best fitting value of alpha= -1.33 +- 0.03 for the combined bands. There is a clear trend of alpha to steepen with increasing wavelength: alpha_(UV & u)=-1.07 +- 0.04 -> alpha_(g & B)=-1.25 +- 0.03 -> alpha_(r & i & z)=-1.33 +- 0.03. We show that the wavelength dependence of the LF slope can be explained by the relative contribution of different SED-type LFs to the overall LF, as different SED types dominate the LF in the blue and red bands. Furthermore we also derive and analyze the luminosity density evolution of the different SED types up to z ~ 2. Based on the FDF data, we find only a mild brightening of M_star and decrease of phi_star with increasing redshift. Therefore, from <z> ~ 0.5 to <z> ~ 3 the characteristic luminosity increases by ~0.8, ~0.4 and ~0.4 magnitudes in the r, i, and z bands, respectively. Simultaneously the characteristic density decreases by about 40 % in all analyzed wavebands. [abridged]