VAGO solutions SauerkrautLM Phi 3 medium Introducing SauerkrautLM Phi 3 medium – our Sauerkraut version of the powerful unsloth/Phi 3 medium 4k instruct! Aligned with DPO using Spectrum QLoRA (by Eric Hartford, Lucas Atkins, Fernando Fernandes Neto and David Golchinfar) targeting 50% of the layers. Table of Contents 1. Overview of all SauerkrautLM Phi 3 medium 2. Model Details Training procedure 3. Evaluation 5. Disclaimer 6. Contact 7. Collaborations 8. Acknowledgement All SauerkrautLM Phi 3 medium Model HF EXL2 GGUF AWQ SauerkrautLM Phi 3 medium Link coming soon coming soon coming soon Model Details SauerkrautLM Phi 3 medium Model Type: SauerkrautLM Phi 3 medium is a finetuned Model based on unsloth/Phi 3 medium 4k instruct Language(s): German, English License: MIT Contact: VAGO solutions Training procedure: We trained this model with Spectrum QLoRA DPO Fine Tuning for 1 epoch with 70k samples targeting 50% of the layers with a high Learningrate of 5e 04. This relatively high learning rate was feasible due to the selective targeting of layers; had we applied this rate to all layers, the gradients would have exploded. Fine Tuning Details Epochs: 1 Data Size: 70,000 samples Targete…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy