Model Details This model is a int4 model with group size 128 of Qwen/Qwen3.5 397B A17B generated by intel/auto round. Please follow the license of the original model. The model is quantized via RTN mode vllm Infernece Example ~~~bash pip install git+https://github.com/vllm project/vllm.git@main pip install git+https://github.com/huggingface/transformers.git ~~~ ~~~bash vllm serve Intel/Qwen3.5 397B A17B int4 AutoRound port 8000 tensor parallel size 1 max model len 2048 reasoning parser qwen3 served model name qwen ~~~ ~~~bash curl http://localhost:8000/v1/chat/completions H "Content Type: application/json" d ' { "model": "qwen", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Summarize Qwen 3.5 in one sentence."} ], "temperature": 1, "max tokens": 512 } ' ~~~ Transformers Inference Transformers v5.2.0 auto round and This pr is required https://github.com/intel/auto round/pull/1476 ~~~python from transformers import AutoProcessor, Qwen3 5MoeForConditionalGeneration model name = "Intel/Qwen3.5 397B A17B int4 AutoRound" model = Qwen3 5MoeForConditionalGeneration.from pretrained(model name, dtype="auto", device map="auto") proce…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy