The gluon polarisation in the nucleon was measured using open charm production by scattering 160 GeV/c polarised muons off longitudinally polarised protons or deuterons. The data were taken by the COMPASS collaboration between 2002 and 2007. A detailed account is given of the analysis method that includes the application of neural networks. Several decay channels of D^0 mesons are investigated. Longitudinal spin asymmetries of the D meson production cross-sections are extracted in bins of D^0 transverse momentum and energy. At leading order QCD accuracy the average gluon polarisation is determined as (Delta g/G)^LO=-0.06 +/- 0.21 (stat.) +/- 0.08 (syst.) at the scale <mu^2> ~13 (GeV/c)^2 and an average gluon momentum fraction <x>~ 0.11. For the first time, the average gluon polarisation is also obtained at next-to-leading order QCD accuracy as (Delta g/G)^NLO = -0.13 +/- 0.15 (stat.) +/- 0.15 (syst.) at the scale <mu^2> ~ 13 (GeV/c)^2 and <x> ~ 0.20.