The asymmetric quantum Rabi model (AQRM) has a broken $mathbb{Z}_2$ symmetry, with generally a non-degenerate eigenvalue spectrum. In some special cases where the asymmetric parameter is a multiple of the cavity frequency, stable level crossings typical of the $mathbb{Z}_2$-symmetric quantum Rabi model are recovered, however, without any obvious parity-like symmetry. This unknown symmetry has thus been referred to as hidden symmetry in the literature. Here we show that this hidden symmetry is not limited to the AQRM, but exists in various related light-matter interaction models with an asymmetric qubit bias term. Conditions under which the hidden symmetry exists in these models are determined and discussed. By investigating tunnelling dynamics in the displaced oscillator basis, a strong connection is found between the hidden symmetry and selective tunnelling.