Context. Distant galaxy clusters provide an effective laboratory in which to study galaxy evolution in dense environments and at early cosmic times. Aims. We aim to identify distant galaxy clusters as extended X-ray sources coincident with overdensities of characteristically bright galaxies. Methods. We use optical and near-infrared (NIR) data from the Hyper Suprime-Cam (HSC) and VISTA Deep Extragalactic Observations (VIDEO) surveys to identify distant galaxy clusters as overdensities of bright, $z_{phot}geq 0.8$ galaxies associated with extended X-ray sources detected in the ultimate XMM extragalactic survey (XXL). Results. We identify a sample of 35 candidate clusters at $0.80leq zleq 1.93$ from an approximately 4.5 deg$^2$ sky area. This sample includes 15 newly discovered candidate clusters, ten previously detected but unconfirmed clusters, and ten spectroscopically confirmed clusters. Although these clusters host galaxy populations that display a wide variety of quenching levels, they exhibit well-defined relations between quenching, cluster-centric distance, and galaxy luminosity. The brightest cluster galaxies (BCGs) within our sample display colours consistent with a bimodal population composed of an old and red subsample together with a bluer, more diverse subsample. Conclusions. The relation between galaxy masses and quenching seem to be already in place at $zsim 1$, although there is no significant variation of the quenching fraction with the cluster-centric radius. The BCG bimodality might be explained by the presence of a younger stellar component in some BCGs but additional data are needed to confirm this scenario.