Model Details This model is a mixed int4 model with group size 128 of Qwen/Qwen3.6 35B A3B generated by intel/auto round. Please follow the license of the original model. Some users have reported an infinite loop issue in our INT4 version. We have added fallbacks for certain layers in this release, but it is still unclear whether the issue has been fully resolved. Please use it with caution. The main branch is generated by auto round best and revision 54e7cd36d9f7a358b4b42740c3aef755638e5ea6 is generated by auto round . Tipically auto round best is better vllm Infernece Example verified on 0.19 ~~~bash vllm serve Intel/Qwen3.6 35B A3B int4 AutoRound port 8000 tensor parallel size 1 max model len 2048 reasoning parser qwen3 served model name qwen speculative config '{"method":"qwen3 next mtp","num speculative tokens":2}' ~~~ ~~~bash curl http://localhost:8000/v1/chat/completions H "Content Type: application/json" d ' { "model": "qwen", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Summarize Qwen 3.6 in one sentence."} ], "temperature": 1, "max tokens": 512 } ' ~~~ Transformers Inference GPTQMODEL<7.0 ~~~python from transfor…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy