Ultrafast pump-probe technique is a powerful tool to understand and manipulate properties of materials for designing novel quantum devices. An intense, single cycle terahertz pulse can change the intrinsic properties of semiconductor quantum dots to have different luminescence. In a hybrid system of plasmon and exciton, the coherence and coupling between these two degrees of freedom play an important role on their optical properties. Therefore, we consider a terahertz pump optical probe experiment in the hybrid systems where the terahertz pump pulse couples to the exciton degrees of freedom on the quantum dot. The time resolved photoluminescence of the hybrid system shows that the response of the characteristic frequency shifts according to the overlap between the pump and probe pulses. Furthermore, the resonance between the exciton and plasmons can be induced by the terahertz pump pulse in some parameter regimes. Our results show the terahertz driven hybrid system can be a versatile tool for manipulating the material properties and open a new route to design modern optical devices.