Video Summary

The paper proposes a new method for training language models, called ``Sparse Transformer''. The method is based on the observation that most of the parameters in a Transformer model are not used during training. The authors propose to sparsify the model by removing the unused parameters, which reduces the computational cost and improves the performance.