Qwen3.5 9B FP8 dynamic Model Overview Model Architecture: Qwen/Qwen3.5 9B Input: Text / Image Output: Text Model Optimizations: Weight quantization: FP8 Activation quantization: FP8 Model size: 14.0 GB (reduced from 19.3 GB in BF16) Release Date: 2026 05 11 Version: 1.0 Model Developers: RedHatAI This model is a quantized version of Qwen/Qwen3.5 9B. Evaluation results and reproduction steps are provided below. Model Optimizations This model was obtained by quantizing the weights and activations of Qwen/Qwen3.5 9B to FP8 data type, ready for inference with vLLM. This optimization reduces the model weights from 19.3 GB to 14.0 GB on disk (~27% reduction). Activations are quantized dynamically at inference time using per tensor scaling, requiring no calibration data. Only the weights and activations of the linear operators within transformer blocks are quantized using LLM Compressor. Deployment Use with vLLM 1. Initialize vLLM server: Multimodal (vision + text): Text only (lower memory): 2. Send requests to the server: Creation This model was created by applying LLM Compressor using data free FP8 dynamic quantization, as presented in the code snippet below. Package versions llm compre…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy