TIPSv2 — SO400m/14 TIPSv2 (Text Image Pre training with Spatial awareness) is a family of contrastive vision language models that produce spatially rich image features aligned with text embeddings. This is the SO400m variant with 412M vision params and 448M text params. Try the code snippets below or check out the GitHub repo for more use cases and visualizations, including zero shot segmentation. Variant Vision params Text params Embed dim DPT Heads B/14 86M 110M 768 B/14 dpt L/14 303M 184M 1024 L/14 dpt SO400m/14 412M 448M 1152 SO400m/14 dpt g/14 1.1B 389M 1536 g/14 dpt Usage Load the model Encode images Images should be tensors in [0, 1] range (just ToTensor() , no ImageNet normalization). Encode text Zero shot classification Visualize spatial features GPU inference Model details Architecture : ViT vision encoder (27 layers) + Transformer text encoder (27 layers) Image preprocessing : resize to any resolution, convert to [0, 1] (no ImageNet normalization) Text preprocessing : SentencePiece tokenizer, lowercased, max 64 tokens Patch size : 14x14 pixels License Apache 2.0 Citation
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy