Mini GLM 4 MoE (0.5B) A small GLM 4 MoE model (543M parameters) for testing and development. Uses the same Glm4MoeForCausalLM architecture as the full GLM 4 100B A10B but with reduced dimensions. This model is designed for testing MoE training pipelines in prime rl without needing large pretrained checkpoints. It is small enough to run on a single GPU while exercising the same code paths as production models. Architecture Parameter Value Parameters 543M Hidden size 1024 Layers 24 Attention heads 16 (4 KV heads) Routed experts 8 Experts per token 4 Shared experts 1 MoE intermediate size 256 Dense intermediate size 2048 Dense layers (first k) 1 Vocab size 151,552 Partial rotary factor 0.5 Model type glm4 moe The architecture mirrors THUDM/GLM 4 100B A10B: the first layer is a dense MLP, and all subsequent layers use Mixture of Experts with a shared expert. Attention uses Grouped Query Attention (GQA) with partial rotary embeddings. How this model was created Step 1: Random initialization. A Glm4MoeConfig was instantiated with the small dimensions above and the HuggingFace Glm4MoeForCausalLM model was initialized with random weights. The tokenizer was copied from THUDM/GLM 4 9B 0414.…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy