We present spectro-photometry spanning 1-5 $mu$m of 51 Eridani b, a 2-10 M$_text{Jup}$ planet discovered by the Gemini Planet Imager Exoplanet Survey. In this study, we present new $K1$ (1.90-2.19 $mu$m) and $K2$ (2.10-2.40 $mu$m) spectra taken with the Gemini Planet Imager as well as an updated $L_P$ (3.76 $mu$m) and new $M_S$ (4.67 $mu$m) photometry from the NIRC2 Narrow camera. The new data were combined with $J$ (1.13-1.35 $mu$m) and $H$ (1.50-1.80 $mu$m) spectra from the discovery epoch with the goal of better characterizing the planet properties. 51 Eri b photometry is redder than field brown dwarfs as well as known young T-dwarfs with similar spectral type (between T4-T8) and we propose that 51 Eri b might be in the process of undergoing the transition from L-type to T-type. We used two complementary atmosphere model grids including either deep iron/silicate clouds or sulfide/salt clouds in the photosphere, spanning a range of cloud properties, including fully cloudy, cloud free and patchy/intermediate opacity clouds. Model fits suggest that 51 Eri b has an effective temperature ranging between 605-737 K, a solar metallicity, a surface gravity of $log$(g) = 3.5-4.0 dex, and the atmosphere requires a patchy cloud atmosphere to model the SED. From the model atmospheres, we infer a luminosity for the planet of -5.83 to -5.93 ($log L/L_{odot}$), leaving 51 Eri b in the unique position as being one of the only directly imaged planet consistent with having formed via cold-start scenario. Comparisons of the planet SED against warm-start models indicates that the planet luminosity is best reproduced by a planet formed via core accretion with a core mass between 15 and 127 M$_{oplus}$.