This paper reports the detailed noise characterization, investigation of various noise sources and its mitigation to improve the performance of a cryogenic bolometer detector. The noise spectrum has been measured for a sapphire bolometer test setup with indigenously developed NTD Ge sensor in the CFDR system at Mumbai. The effect of external noise, arising either from ground loops in the system or from the diagnostic and control electronics of the cryostat, on the performance of a cryogenic bolometer is assessed. A systematic comparison of the influence of different noise pickups on the bolometer resolution is also presented. The best-achieved resolution at 15mK is ~15 keV for heater pulses and appears to be mainly limited by the noise due to the pulse tube cryocooler.