Dedicated to building a more intuitive, comprehensive, and efficient LLMs compression toolkit. 📖 Documentation      🤗 Hugging Face      🤖 ModelScope      💬 WeChat Hy3 llama.cpp Quantization Low bit quantized Hy3 (hy v3) GGUFs for llama.cpp — ready to run out of the box, with MTP self speculative decoding. This repo ships the quantized models, plus the mixed precision recipes to build your own from a calibration set. Two parts below: Deploy (build & run) and Quantization . Quick Start Build Clone the latest llama.cpp (must include the hy v3 support merged in PR 25395 — i.e. any commit after 505b1ed ; the current master is fine) and build it following the official build guide: Binaries land in llama.cpp hyv3/build/bin/ . Already downloaded an older GGUF? Earlier GGUFs here were built against a patched llama.cpp and won't load correctly on today's upstream. To build and run those, follow the previous README (the setup hyv3 llama.sh + patches flow) instead. Run Recommended setups GPUs build MTP c (context) KV cache 1× H20 (96 GB) IQ1 M no c 65536 ctk q8 0 ctv q8 0 2× H20 (192 GB) IQ1 M yes / (default) / (f16) 2× H20 (192 GB) Q4 K M ye…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy