We point out that, if the direct CP asymmetries in the $D to pi^+ pi^-$ and $D to K^+ K^-$ decays are unequal, the indirect CP asymmetries as measured in these modes are necessarily unequal. This nonuniversality of indirect CP asymmetries can be significant with the right amount of new physics contributions, a scenario that may be fine-tuned, but is still viable. A model-independent fit to the current data allows different indirect CP asymmetries in the above two decays. This could even be contributing to the apparent tension between the difference CP asymmetries $Delta A_{rm CP}$ measured through the pion-tagged and muon-tagged data samples at the LHCb. This also implies that the measurements of $A_Gamma$ and $y_{rm CP}$ in the $pi^+ pi^-$ and $K^+ K^-$ decay modes can be different, and averaging over these two modes should be avoided. In any case, the complete analysis of CP violation measurements in the $D$ meson sector needs to take into account the possibility of different indirect CP asymmetries in the $pi^+pi^-$ and $K^+ K^-$ channels.