We have carried out extensive comparative studies of the structural and transport properties of CaRuO$_3$ thin films grown under various oxygen pressure. We find that the preferred orientation and surface roughness of the films are strongly affected by the oxygen partial pressure during growth. This in turn affects the electrical and magnetic properties of the films. Films grown under high oxygen pressure have the least surface roughness and show transport characteristics of a good metal down to the lowest temperature measured. On the other hand, films grown under low oxygen pressures have high degree of surface roughness and show signatures of ferromagnetism. We could verify that the low frequency resistance fluctuations (noise) in these films arise due to thermally activated fluctuations of local defects and that the defect density matches with the level of disorder seen in the films through structural characterizations.