Recently, knowledge distillation (KD) has shown great success in BERT compression. Instead of only learning from the teachers soft label as in conventional KD, researchers find that the rich information contained in the hidden layers of BERT is conducive to the students performance. To better exploit the hidden knowledge, a common practice is to force the student to deeply mimic the teachers hidden states of all the tokens in a layer-wise manner. In this paper, however, we observe that although distilling the teachers hidden state knowledge (HSK) is helpful, the performance gain (marginal utility) diminishes quickly as more HSK is distilled. To understand this effect, we conduct a series of analysis. Specifically, we divide the HSK of BERT into three dimensions, namely depth, length and width. We first investigate a variety of strategies to extract crucial knowledge for each single dimension and then jointly compress the three dimensions. In this way, we show that 1) the students performance can be improved by extracting and distilling the crucial HSK, and 2) using a tiny fraction of HSK can achieve the same performance as extensive HSK distillation. Based on the second finding, we further propose an efficient KD paradigm to compress BERT, which does not require loading the teacher during the training of student. For two kinds of student models and computing devices, the proposed KD paradigm gives rise to training speedup of 2.7x ~ 3.4x.
Download