Recent research has been constraining the retention fraction of black holes (BHs) in globular clusters by comparing the degree of mass segregation with $N$-body simulations. They are consistent with an upper limit of the retention fraction being $50,%$ or less. In this work, we focus on direct simulations of the dynamics of BHs in star clusters. We aim to constrain the effective distribution of natal kicks that BHs receive during supernova (SN) explosions and to estimate the BH retention fraction. We used the collisional $N$-body code nbody6 to measure the retention fraction of BHs for a given set of parameters, which are: the initial mass of a star cluster, the initial half-mass radius, and $sigma_mathrm{BH}$, which sets the effective Maxwellian BH velocity kick distribution. We compare these direct $N$-body models with our analytic estimates and newest observational constraints. The numerical simulations show that for the one-dimensional (1D) velocity kick dispersion $sigma_mathrm{BH} < 50,mathrm{km,s^{-1}}$, clusters with radii of 2 pc and that are initially more massive than $5 times 10^3,M_odot$ retain more than $20,%$ of BHs within their half-mass radii. Our simple analytic model yields a number of retained BHs that is in good agreement with the $N$-body models. Furthermore, the analytic estimates show that ultra-compact dwarf galaxies (UCDs) should have retained more than $80,%$ of their BHs for $sigma_mathrm{BH} leq 190,mathrm{km,s^{-1}}$. Although our models do not contain primordial binaries, in the most compact clusters with $10^3$ stars, we have found evidence of delayed SN explosions producing a surplus of BHs compared to the IMF due to dynamically formed binary stars. These cases do not occur in the more populous or expanded clusters.