A useful concept in the development of physical models on the $kappa$-Minkowski noncommutative spacetime is that of a curved momentum space. This structure is not unique: several inequivalent momentum space geometries have been identified. Some are associated to a different assumption regarding the signature of spacetime (i.e. Lorentzian vs. Euclidean), but there are inequivalent momentum spaces that can be associated to the same signature and even the same group of symmetries. Moreover, in the literature there are two approaches to the definition of these momentum spaces, one based on the right- (or left-)invariant metrics on the Lie group generated by the $kappa$-Minkowski algebra. The other is based on the construction of $5$-dimensional matrix representation of the $kappa$-Minkowski coordinate algebra. Neither approach leads to a unique construction. Here, we find the relation between these two approaches and introduce a unified approach, capable of describing all momentum spaces, and identify the corresponding quantum group of spacetime symmetries. We reproduce known results and get a few new ones. In particular, we describe the three momentum spaces associated to the $kappa$-Poincare group, which are half of a de Sitter, anti-de Sitter or Minkowski space, and we identify what distinguishes them. Moreover, we find a new momentum space with the geometry of a light cone, associated to a $kappa$-deformation of the Carroll group.