It has been well established experimentally that the interplay of electronic correlations and spin-orbit interactions in Ir$^{4+}$ and Ir$^{5+}$ oxides results in insulating J$_{rm eff}$=1/2 and J$_{rm eff}$=0 ground states, respectively. However, in compounds where the structural dimerization of iridum ions is favourable, the direct Ir $d$--$d$ hybridisation can be significant and takes a key role. Here, we investigate the effects of direct Ir $d$--$d$ hybridisation in comparison with electronic correlations and spin-orbit coupling in Ba$_5$AlIr$_2$O$_{11}$, a compound with Ir dimers. Using a combination of $ab$ $initio$ many-body wave function quantum chemistry calculations and resonant inelastic X-ray scattering (RIXS) experiments, we elucidate the electronic structure of Ba$_5$AlIr$_2$O$_{11}$. We find excellent agreement between the calculated and the measured spin-orbit excitations. Contrary to the expectations, the analysis of the many-body wave function shows that the two Ir (Ir$^{4+}$ and Ir$^{5+}$) ions in the Ir$_2$O$_9$ dimer unit in this compound preserve their local J$_{rm eff}$ character close to 1/2 and 0, respectively. The local point group symmetry at each of the Ir sites assumes an important role, significantly limiting the direct $d$--$d$ hybridisation. Our results emphasize that minute details in the local crystal field (CF) environment can lead to dramatic differences in electronic states in iridates and 5$d$ oxides in general.