Qwen3.5-27B-W4A16-AWQ-HD-Agent

This is a highly optimized, hybrid W4A16 AWQ quantization of Qwen/Qwen3.5-27B. It was built using a custom llmcompressor pipeline designed to preserve maximum reasoning, agentic logic, and structural integrity.

🎯 Hybrid AWQ Quantization Strategy

Unlike standard uniform 4-bit AWQ this model uses a surgical, hybrid-precision recipe. Crucial structural and recurrent layers were explicitly bypassed to prevent the logic degradation and multimodal failure typical of heavy quantization:

• Protected Layers (16-bit): lm_head, embed_tokens, linear_attn (recurrent state preservation), vision towers (model.visual.*), and next-token prediction blocks (mtp.*).

• Targeted Smoothing: AWQ scaling was selectively applied using a custom mapped layer balance (smoothing the input layernorm of every 4th attention layer, and all MLP blocks universally) to maintain coherence across extended generations.

• Weight Precision: 4-bit INT

• Group Size: 128

📚 High-Depth Calibration Mixture

To anchor the model's performance for agentic workflows, the calibration process utilized a precisely balanced 1,024-sample dataset at a 4096 sequence length. The mixture is heavily skewed toward reasoning traces, complex system prompts, and structured tool-use.

• [lambda/hermes-agent-reasoning-traces] (500 samples): Provides the core foundation for tool-use, multi-step deduction, and structured agent trajectories.

• [HuggingFaceH4/ultrachat_200k] (300 samples): Anchors multi-turn conversational flow and general instruction following.

• [Open-Orca/OpenOrca] (150 samples): Reinforces strict system prompt adherence and analytical breakdown.

• [Salesforce/wikitext] (74 samples): Stabilizes baseline perplexity and general linguistic coherence.

⚠️ Critical Generation Settings

To prevent the model from getting stuck in endless thinking/reasoning loops, it is highly recommended to override the default generation config with the following sampler settings:

• Temperature: 0.75

• Top P: 0.95

• Top K: 20

• Presence Penalty: 0.3

• Repetition Penalty: 1.05

• Frequency Penalty: 1.15

📚 Sample vLLM/Docker Compose Config

environment:
  - VLLM_WORKER_MULTIPROC_METHOD=spawn
  - ATTENTION_BACKEND=FLASHINFER
  - VLLM_DISABLED_KERNELS=AllSparkLinearKernel
  - DISABLE_CUSTOM_ALL_REDUCE=1
  - MAX_MODEL_LEN=auto
  - OPT_LEVEL=3
  - MAMBA_CACHE_MODE=all
  - MAMBA_BLOCK_SIZE=8
  - COMPILATION_CONFIG={"mode":"VLLM_COMPILE","cudagraph_capture_sizes":[4,8,12]}
  - KERNEL_CONFIG={"enable_flashinfer_autotune":false}
  - VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
  - VLLM_MARLIN_USE_ATOMIC_ADD=1
  - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
command: >
  --model ekozzer/Qwen3.5-27B-W4A16-AWQ-HD-Agent-v3
  --quantization compressed-tensors
  --kv-cache-dtype fp8_e4m3
  --block-size 16
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'
  --language-model-only
  --generation-config auto
  --override-generation-config '{"temperature":0.75,"top_p":0.95,"top_k":20,"presence_penalty":0.3,"repetition_penalty":1.05,"frequency_penalty":1.15}'
  --dtype bfloat16
  --default-chat-template-kwargs '{"enable_thinking": true}'
  --max-num-seqs 2
  --trust-remote-code 
  --enable-auto-tool-choice
  --tool-call-parser qwen3_coder
  --reasoning-parser qwen3
  --enforce-eager

Note

I am also running in a customized docker container using the following dockerfile settings

# Use the official vLLM base image
FROM vllm/vllm-openai:v0.22.1 as base

# Install git and other utilities
RUN apt-get update && apt-get install -y git && rm -rf /var/lib/apt/lists/*

# Install the Conch kernel extensions cleanly
RUN pip install conch-triton-kernels