Qwen3 1.6B A0.9B This is a tiny version of Qwen/Qwen3 30B A3B created for testing and development. Model Details Base Model : Qwen/Qwen3 30B A3B Architecture : qwen3 moe (Mixture of Experts) Total Parameters : 1.57B Activated Parameters : ~0.9B (50% MoE activation) Configuration Changes The following parameters were reduced from the original model: Parameter Original Tiny num hidden layers 48 10 num local experts 128 16 num experts per tok 8 8 hidden size 2048 2048 intermediate size 6144 6144 moe intermediate size 768 768 num attention heads 32 32 num key value heads 4 4 Checkpoint Structure The checkpoint is stored as a single model.safetensors file with individual expert weights matching the original Qwen3 structure. Each layer has 16 experts with separate gate proj , up proj , and down proj weights per expert. Validation The model was fine tuned on a toy copypasta dataset and achieves: Perplexity : 1.0 (on validation text) Generation : Successfully generates coherent continuations Example generation: Usage Creation Process This model was created using the llm compressor create tiny model Claude skill: 1. Configuration : Created with 10 layers and 16 experts (8 activated per toke…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy