codelion/Qwen3 0.6B accuracy recovery lora 🎯 Accuracy Recovery LoRA Adapter This LoRA adapter helps recover accuracy when using INT4 quantized versions of Qwen/Qwen3 0.6B. It was trained using self distillation with Magpie generated data. 📊 Performance Metrics Base Model : Qwen/Qwen3 0.6B Quantization : INT4 with NF4 LoRA Rank : 64 LoRA Alpha : 128 Training Samples : 610 Target Performance Gap : <5% perplexity increase 🔧 Usage 🧪 Training Details Method : Self distillation using Magpie data generation Framework : PEFT + LoRA Loss Function : Combined KL divergence + MSE loss Temperature : 1.0 Alpha (distillation weight) : 0.01 📈 Expected Benefits ✅ Maintains accuracy close to FP16 baseline ✅ ~75% reduction in memory usage ✅ 2 3x faster inference than FP16 ✅ Easy to integrate with existing workflows 🏷️ Related Dataset : codelion/Qwen3 0.6B magpie Base Model : Qwen/Qwen3 0.6B Framework : PEFT This adapter is part of the Ellora project standardized recipes for enhancing LLM capabilities.
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy