Background

Vision transformers are now seen as the backbone for various vision tasks.

Training data-efficient image transformers & distillation through attention

DeiT III: Revenge of the ViT

ICCV 2021 Open Access Repository

PVT v2: Improved Baselines with Pyramid Vision Transformer

Knowledge Distillation

General KD

Distilling the Knowledge in a Neural Network

• The very very early KD paper by Hinton, method is old, but useful as a citation

Fitnets: Hints for thin deep nets

KD on Language models

TinyBERT: Distilling BERT for Natural Language Understanding

• Three level KD at post-norm (MSE), attention (MSE) and output distribution (CrossEntropy)

TernaryBERT: Distillation-aware Ultra-low Bit BERT

• Very similar to TinyBERT but applied to quantization. The quantization they used contained a ’scaling factor’.

EBERT: Efficient BERT Inference with Dynamic Structured Pruning

• KD on FFN and attention

Styles of KD

Kernel Based Progressive Distillation for Adder Neural Networks