Nandi Mini 600M Early Checkpoint Introduction Nandi Mini 600M Early Checkpoint is an early stage checkpoint (After 250 Billions tokens ) from the upcoming Nandi Mini 600M model family, this is not the final model , a compact multilingual language model focused on strong efficiency, deployment flexibility, and Indic language support. The model is being trained completely from scratch and is designed to deliver strong performance at low compute and memory budgets. This checkpoint is shared to provide an early look into the model’s scaling behavior and training progress. This release is an early checkpoint and not the final converged model. Performance is expected to improve further with continued training and scaling. 📢 We will soon share technical blog ! Stay tuned! Architectural Highlights Nandi Mini 600M introduces several efficiency focused architectural optimizations designed for compact yet capable language models. Shared KV (Shared Key Value Vectors) Shared KV is one of the core architectural ideas explored in Nandi Mini. Instead of computing separate Key and Value projections, both reuse a shared latent representation, while a lightweight Key normalization step is applied sp…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy