TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance [ICCV 2023] TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance TinyCLIP is a novel cross modal distillation method for large scale language image pre trained models. The method introduces two core techniques: affinity mimicking and weight inheritance . This work unleashes the capacity of small CLIP models, fully leveraging large scale models as well as pre training data and striking the best trade off between speed and accuracy. Use with Transformers Highlights TinyCLIP ViT 45M/32 uses only half parameters of ViT B/32 to achieves comparable zero shot performance . TinyCLIP ResNet 19M reduces the parameters by 50\% while getting 2x inference speedup, and obtains 56.4\% accuracy on ImageNet. Model Zoo Model Weight inheritance Pretrain IN 1K Acc@1(%) MACs(G) Throughput(pairs/s) Link TinyCLIP ViT 39M/16 Text 19M manual YFCC 15M 63.5 9.5 1,469 Model TinyCLIP ViT 8M/16 Text 3M manual YFCC 15M 41.1 2.0 4,150 Model TinyCLIP ResNet 30M Text 29M manual LAION 400M 59.1 6.9 1,811 Model TinyCLIP ResNet 19M Text 19M manual LAION 400M 56.4 4.4 3,024 Model TinyCLIP ViT 61M/32 Text 29M manual LAION 400M…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy