Model Details This model is a mixed int4 model with group size 128 and symmetric quantization of Qwen/Qwen3 Omni 30B A3B Instruct generated by intel/auto round. Please follow the license of the original model. vllm omni inference Setup ~~~bash pip install git+https://github.com/lvliang intel/vllm omni.git@feats/ar w4a16 qwen omni pip install git+https://github.com/huggingface/transformers.git ~~~ ~~~bash CUDA VISIBLE DEVICES=0,1 vllm serve Intel/Qwen3 Omni 30B A3B Instruct int4 AutoRound omni port 8091 curl s http://localhost:8091/v1/chat/completions \ H "Content Type: application/json" \ d '{ "model": "Intel/Qwen3 Omni 30B A3B Instruct int4 AutoRound", "messages": [ { "role": "system", "content": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech." }, { "role": "user", "content": "What is 2 + 3? Answer with just the number." } ], "temperature": 0.0, "max tokens": 2048, "repetition penalty": 1.1 }' curl s http://127.0.0.1:8091/v1/chat/completions \ H "Content Type: application/json" \ d '{ "model": "Intel/Qwen3 Omni 30B A3B Instruct int4 AutoRound", "messages": [ { "role":…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy