Rhombohedral $N = 3$ trilayer graphene on hexagonal boron nitride (TLG/BN) hosts gate-tunable, valley-contrasting, nearly flat topological bands that can trigger spontaneous quantum Hall phases under appropriate conditions of the valley and spin polarization. Recent experiments have shown signatures of C = 2 valley Chern bands at 1/4 hole filling, in contrast to the predicted value of C = 3. We discuss the low-energy model for rhombohedral N-layer graphene (N = 1, 2, 3) aligned with hexagonal boron nitride (hBN) subject to off-diagonal moire vector potential terms that can alter the valley Chern numbers. Our analysis suggests that topological phase transitions of the flat bands can be triggered by pseudomagnetic vector field potentials associated to moire strain patterns, and that a nematic order with broken rotational symmetry can lead to valley Chern numbers that are in agreement with recent Hall conductivity observations.