In this paper, we present a robust multipath-based localization and mapping framework that exploits the phases of specular multipath components (MPCs) using a massive multiple-input multiple-output (MIMO) array at the base station. Utilizing the phase information related to the propagation distances of the MPCs enables the possibility of localization with extraordinary accuracy even with limited bandwidth. The specular MPC parameters along with the parameters of the noise and the dense multipath component (DMC) are tracked using an extended Kalman filter (EKF), which enables to preserve the distance-related phase changes of the MPC complex amplitudes. The DMC comprises all non-resolvable MPCs, which occur due to finite measurement aperture. The estimation of the DMC parameters enhances the estimation quality of the specular MPCs and therefore also the quality of localization and mapping. The estimated MPC propagation distances are subsequently used as input to a distance-based localization and mapping algorithm. This algorithm does not need prior knowledge about the surrounding environment and base station position. The performance is demonstrated with real radio-channel measurements using an antenna array with 128 ports at the base station side and a standard cellular signal bandwidth of 40 MHz. The results show that high accuracy localization is possible even with such a low bandwidth.