The appearance of half-quantized thermal Hall conductivity in $alpha$-RuCl$_3$ in the presence of in-plane magnetic fields has been taken as a strong evidence for Kitaev spin liquid. Apart from the quantization, the observed sign structure of the thermal Hall conductivity is also consistent with predictions from the exact solution of the Kitaev model. Namely, the thermal Hall conductivity changes sign when the field direction is reversed with respect to the heat current, which is perpendicular to one of the three nearest neighbor bonds on the honeycomb lattice. On the other hand, it is almost zero when the field is applied along the bond direction. Here, we show that such a peculiar sign structure of the thermal Hall conductivity is a generic property of the polarized state in the presence of in-plane magnetic-fields. In this case, thermal Hall effect arises from topological magnons with finite Chern numbers and the sign structure follows from the symmetries of the momentum space Berry curvature. Using a realistic spin model with bond-dependent interactions, we show that the thermal Hall conductivity can have a magnitude comparable to that observed in the experiments. Hence the sign structure alone cannot make a strong case for Kitaev spin liquid. The quantization at very low temperatures, however, will be a decisive test as the magnon contribution vanishes in the zero temperature limit.