We present a detailed derivation of the observed galaxy number over-density on cosmological scales up to second order in perturbation theory. We include all relativistic effects that arise from observing on the past lightcone. The derivation is in a general gauge, and applies to all dark energy models (including interacting dark energy) and many modified gravity models. The result will be important for accurate cosmological parameter estimation, including non-Gaussianity, since all projection effects need to be taken into account. It also offers the potential for new probes of General Relativity, dark energy and modified gravity. This paper accompanies Paper I which presents the key results for the concordance model in Poisson gauge.