The near-Earth asteroid (308635) 2005 YU55 is a potentially hazardous asteroid which was discovered in 2005 and passed Earth on November 8th 2011 at 0.85 lunar distances. This was the closest known approach by an asteroid of several hundred metre diameter since 1976 when a similar size object passed at 0.5 lunar distances. We observed 2005 YU55 from ground with a recently developed mid-IR camera (miniTAO/MAX38) in N- and Q-band and with the Submillimeter Array (SMA) at 1.3 mm. In addition, we obtained space observations with Herschel/PACS at 70, 100, and 160 micron. Our thermal measurements cover a wide range of wavelengths from 8.9 micron to 1.3 mm and were taken after opposition at phase angles between -97 deg and -18 deg. We performed a radiometric analysis via a thermophysical model and combined our derived properties with results from radar, adaptive optics, lightcurve observations, speckle and auxiliary thermal data. We find that (308635) 2005 YU55 has an almost spherical shape with an effective diameter of 300 to 312 m and a geometric albedo pV of 0.055 to 0.075. Its spin-axis is oriented towards celestial directions (lam_ecl, beta_ecl) = (60 deg +/- 30deg, -60 deg +/- 15 deg), which means it has a retrograde sense of rotation. The analysis of all available data combined revealed a discrepancy with the radar-derived size. Our radiometric analysis of the thermal data together with the problem to find a unique rotation period might be connected to a non-principal axis rotation. A low to intermediate level of surface roughness (r.m.s. of surface slopes in the range 0.1 - 0.3) is required to explain the available thermal measurements. We found a thermal inertia in the range 350-800 Jm^-2s^-0.5K^-1, very similar to the rubble-pile asteroid (25143) Itokawa and indicating a mixture of low conductivity fine regolith with larger rocks and boulders of high thermal inertia on the surface.