Motivated in part by the pseudo-Nambu Goldstone Boson mechanism of electroweak symmetry breaking in Composite Higgs Models, in part by dark matter scenarios with strongly coupled origin, as well as by general theoretical considerations related to the large-N extrapolation, we perform lattice studies of the Yang-Mills theories with $Sp(2N)$ gauge groups. We measure the string tension and the mass spectrum of glueballs, extracted from appropriate 2-point correlation functions of operators organised as irreducible representations of the octahedral symmetry group. We perform the continuum extrapolation and study the magnitude of finite-size effects, showing that they are negligible in our calculation. We present new numerical results for $N=1$, $2$, $3$, $4$, combine them with data previously obtained for $N=2$, and extrapolate towards $Nrightarrow infty$. We confirm explicitly the expectation that, as already known for $N=1,2$ also for $N=3,4$ a confining potential rising linearly with the distance binds a static quark to its antiquark. We compare our results to the existing literature on other gauge groups, with particular attention devoted to the large-$N$ limit. We find agreement with the known values of the mass of the $0^{++}$, $0^{++*}$ and $2^{++}$ glueballs obtained taking the large-$N$ limit in the $SU(N)$ groups. In addition, we determine for the first time the mass of some heavier glueball states at finite $N$ in $Sp(2N)$ and extrapolate the results towards $N rightarrow +infty$ taking the limit in the latter groups. Since the large-$N$ limit of $Sp(2N)$ is the same as in $SU(N)$, our results are relevant also for the study of QCD-like theories.