We propose a three-mode optomechanical system to realize optical nonreciprocal transmission with unidirectional amplification, where the system consists of two coupled cavities and one mechanical resonator which interacts with only one of the cavities. Additionally, the optical gain is introduced into the optomechanical cavity. It is found that for a strong optical input, the optical transmission coefficient can be greatly amplified in a particular direction and suppressed in the opposite direction. The expressions of the optimal transmission coefficient and the corresponding isolation ratio are given analytically. Our results pave a way to design high-quality nonreciprocal devices based on optomechanical systems.