Read our guide for detailed instructions on running DeepSeek V3 0324 locally. Unsloth's Dynamic Quants is selectively quantized, greatly improving accuracy over standard bits. DeepSeek V3 0324 Dynamic GGUF Our DeepSeek V3 0324 GGUFs allow you to run the model in llama.cpp, LMStudio, Open WebUI and other inference frameworks. Includes 1 4 bit Dynamic versions, which yields better accuracy and results than standard quantization. MoE Bits Type Disk Size Accuracy Link Details 1.78bit (prelim) IQ1 S 186GB Ok Link down proj in MoE mixture of 2.06/1.78bit 1.93bit (prelim) IQ1 M 196GB Fair Link down proj in MoE mixture of 2.06/1.93bit 2.42bit IQ2 XXS 219GB Recommended Link down proj in MoE all 2.42bit 2.71bit Q2 K XL 248GB Recommended Link down proj in MoE mixture of 3.5/2.71bit 3.5bit Q3 K XL 321GB Great Link down proj in MoE mixture of 4.5/3.5bit 4.5bit Q4 K XL 405GB Best Link down proj in MoE mixture of 5.5/4.5bit Prelim = preliminary through our testing, they're generally fine but sometimes don't produce the best code and so more work/testing needs to be done. 2.71bit was found to be the best in terms of performance/size and produces code that is great and works well. 2.42bit was also…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy