The non-trivial magnon band topology and its consequent responses have been extensively studied in two-dimensional magnetisms. However, the triangular lattice antiferromagnet (TLAF), the best-known frustrated two-dimensional magnet, has received less attention than the closely related Kagome system, because of the spin-chirality cancellation in the umbrella ground state of the undistorted TLAF. In this work, we study the band topology and the thermal Hall effect (THE) of the TLAF with (anti-)trimerization distortion under the external perpendicular magnetic field using the linearized spin wave theory. We show that the spin-chirality cancellation is removed in such case, giving rise to the non-trivial magnon band topology and the finite THE. Moreover, the magnon bands exhibit band topology transitions tuned by the magnetic field. We demonstrate that such transitions are accompanied by the logarithmic divergence of the first derivative of the thermal Hall conductivity. Finally, we examine the above consequences by calculating the THE in the hexagonal manganite YMnO$_3$, well known to have anti-trimerization.