Squeezed states and macroscopic superpositions of coherent states have been predicted to be generated dynamically in Bose Josephson junctions. We solve exactly the quantum dynamics of such a junction in the presence of a classical noise coupled to the population-imbalance number operator (phase noise), accounting for, for example, the experimentally relevant fluctuations of the magnetic field. We calculate the correction to the decay of the visibility induced by the noise in the non-Markovian regime. Furthermore, we predict that such a noise induces an anomalous rate of decoherence among the components of the macroscopic superpositions, which is independent of the total number of atoms, leading to potential interferometric applications.