Moondream 3 (Preview) is an vision language model with a mixture of experts architecture (9B total parameters, 2B active). This model makes no compromises, delivering state of the art visual reasoning while still retaining our efficient and deployment friendly ethos. ✨ Demo & 8201; · & 8201; ☁️ Cloud API & 8201; · & 8201; 📝 Release notes Architecture 1. 24 layers; the first four are dense, the rest have MoE FFNs with 64 experts, 8 activated per token 2. MoE FFNs have GeGLU architecture, with inner/gate dim of 1024. The model's hidden dim is 2048. 3. Usable context length increased to 32K, with a custom efficient SuperBPE tokenizer 4. Multi headed attention with learned position and data dependent temperature scaling 5. SigLIP based vision encoder, with multi crop channel concatenation for token efficient high resolution image processing For more details, please refer to the release notes. Or try the model out in our playground demo. The following instructions demonstrate how to run the model locally using Transformers. We also offer a cloud API with a generous free tier that can help you get started quicker! Usage Load the model and prepare it for inference. We use FlexAttention f…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy