We developed an apparatus to couple a 50-micrometer diameter whispering-gallery silica microtoroidal resonator in a helium-4 cryostat using a straight optical tapered-fiber at 1550nm wavelength. On a top-loading probe specifically adapted for increased mechanical stability, we use a specifically-developed cryotaper to optically probe the cavity, allowing thus to record the calibrated mechanical spectrum of the optomechanical system at low temperatures. We then demonstrate excellent thermalization of a 63-MHz mechanical mode of a toroidal resonator down to the cryostats base temperature of 1.65K, thereby proving the viability of the cryogenic refrigeration via heat conduction through static low-pressure exchange gas. In the context of optomechanics, we therefore provide a versatile and powerful tool with state-of-the-art performances in optical coupling efficiency, mechanical stability and cryogenic cooling.