Llama 3.2 0.5B Instruct This is a tiny version of meta llama/Llama 3.2 1B Instruct created for testing and development. Model Details Base Model : meta llama/Llama 3.2 1B Instruct Architecture : llama Total Parameters : 0.51B Activated Parameters : 0.51B (non MoE) Configuration Changes The following parameters were reduced from the original model: Parameter Original Tiny num hidden layers 16 4 hidden size 2048 2048 intermediate size 8192 8192 num attention heads 32 32 num key value heads 8 8 Checkpoint Structure This model uses a single model.safetensors file containing all weights. The checkpoint structure is identical to the original model, with the standard Llama architecture tensors: model.embed tokens.weight model.layers. .self attn.{q,k,v,o} proj.weight model.layers. .mlp.{gate,up,down} proj.weight model.layers. .{input,post attention} layernorm.weight model.norm.weight Usage Validation Creation Process This model was created using the llm compressor create tiny model claude skill: 1. Inspected the original model configuration to identify key parameters 2. Created a tiny version by reducing num hidden layers from 16 to 4 3. Fine tuned the model on a toy dataset (famous copypa…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy