N-Higgs doublet models (NHDM) are a popular framework to construct electroweak symmetry breaking mechanisms beyond the Standard model. Usually, one builds an NHDM scalar sector which is invariant under a certain symmetry group. Although several such groups have been used, no general analysis of symmetries possible in the NHDM scalar sector exists. Here, we make the first step towards this goal by classifying the elementary building blocks, namely the abelian symmetry groups, with a special emphasis on finite groups. We describe a strategy that identifies all abelian groups which are realizable as symmetry groups of the NHDM Higgs potential. We consider both the groups of Higgs-family transformations only and the groups which also contain generalized CP transformations. We illustrate this strategy with the examples of 3HDM and 4HDM and prove several statements for arbitrary N.