Load modeling is difficult due to its uncertain and time-varying properties. Through the recently proposed ambient signals load modeling approach, these properties can be more frequently tracked. However, the large dataset of load modeling results becomes a new problem. In this paper, a hierarchical temporal and spatial clustering method of load models is proposed, after which the large size load model dataset can be represented by several representative load models (RLMs). In the temporal clustering stage, the RLMs of one load bus are picked up through clustering to represent all the load models of the load bus at different time. In the spatial clustering stage, the RLMs of all the load buses form a new set and the RLMs of the system are picked up through spatial clustering. In this way, the large sets of load models are represented by a small number of RLMs, through which the storage space of the load models is significantly reduced. The validation results in IEEE 39 bus system have shown that the simulation accuracy can still be maintained after replacing the load models with the RLMs. In this way, the effectiveness of the proposed hierarchical clustering framework is validated.