In this work, the thermodynamic properties of the organic superconductor $lambda$-(BETS)$_2$GaCl$_4$ are investigated to study a high-field superconducting state known as the putative Fulde-Ferrell-Larkin-Ovchinnikov (FFLO) phase. We observed a small thermodynamic anomaly in the field $H_{rm FFLO}$ $sim$ 10~T, which corresponds to the Pauli limiting field $H_{rm P}$. This anomaly probably originates from a transition from a uniform superconducting state to the FFLO state. $H_{rm FFLO}$ does not show a strong field-angular dependence due to a quasi-isotropic paramagnetic effect in $lambda$-(BETS)$_2$GaCl$_4$. The thermodynamic anomaly at $H_{rm FFLO}$ is smeared out and low-temperature upper critical field $H_{rm c2}$ changes significantly if fields are not parallel to the conducting plane even for a deviation of $sim$0.5$^{circ}$. This behavior indicates that the high-field state is very unstable, as it is influenced by the strongly anisotropic orbital effect. Our results are consistent with the theoretical predictions on the FFLO state, and show that the high-field superconductivity is probably an FFLO state in $lambda$-(BETS)$_2$GaCl$_4$ from a thermodynamic point of view.