We derive an analytical expression for the two-gluon production in the pA (light-heavy) collisions, and focus specifically on the rapidity dependent part. We approximate the gauge field from the heavy target as the Color Glass Condensate which interacts with the light projectile whose source density allows for a perturbative expansion. We discuss the longitudinal correlations of produced particles. Our calculation goes in part beyond the eikonal limit for the emitted gluons so that we can retain the exponential terms with respect to the rapidity difference. Our expression can thus describe the short-range correlations as well as the long-range ones for which our formula is reduced to the known expression. In a special case of two high-pt gluons in the back-to-back kinematics we find that dependence on the rapidity separation is only moderate even in the diagrammatically connected part.