A controversy arose over the interpretation of the recently observed hump features in Hall resistivity $rho_{xy}$ from ultra-thin SrRuO$_3$ (SRO) film; it was initially interpreted to be due to topological Hall effect but was later proposed to be from existence of regions with different anomalous Hall effect (AHE). In order to settle down the issue, we performed Hall effect as well as magneto-optic Kerr-effect measurements on 4 unit cell SRO films grown on SrTiO$_3$ (001) substrates. Clear hump features are observed in the measured $rho_{xy}$, whereas neither hump feature nor double hysteresis loop is seen in the Kerr rotation which should be proportional to the magnetization. In addition, magnetization measurement by superconducting quantum interference device shows no sign of multiple coercive fields. These results show that inhomogeneous AHE alone cannot explain the observed hump behavior in $rho_{xy}$ data from our SRO ultra-thin films. We found that emergence of the hump structure in $rho_{xy}$ is closely related to the growth condition, high quality films having clear sign of humps.