In contrast to electron (fermion) systems, topological phases of charge neutral bosons have been poorly understood despite recent extensive research on insulating magnets. The most important unresolved issue is how the inevitable inter-bosonic interactions influence the topological properties. It has been proposed that the quantum magnet SrCu$_2$(BO$_3$)$_2$ with an exact ground state serves as an ideal platform for this investigation, as the system is expected to be a magnetic analogue of a Chern insulator with electrons replaced by bosonic magnetic excitations (triplons). Here, in order to examine topologically protected triplon chiral edge modes in SrCu$_2$(BO$_3$)$_2$, we measured the thermal Hall conductivity $kappa_{xy}$ with extremely high accuracy. Contrary to the theoretical expectations, no discernible $kappa_{xy}$ was observed, which is at most $sim$ 1/20 of the prediction if present. This implies that even relatively weak inter-particle interactions seriously influence the topological transport properties at finite temperatures. These demonstrate that, in contrast to fermionic cases, the picture of non-interacting topological quasi-particles cannot be naively applied to bosonic systems, calling special attention to the interpretation of the topological bosonic excitations reported for various insulating magnets.