Mistral Nemo Instruct 2407 Abliterated — AWQ (W4A16) AWQ 4 bit quantization of natong19/Mistral Nemo Instruct 2407 abliterated , itself an abliteration of Mistral Nemo Instruct 2407. Why this exists The base abliterated model was only published in BF16 and GGUF. This AWQ quant is ~7 GB VRAM and serves natively in vLLM with the Marlin kernel for fast inference. Quantization details Tool: llm compressor (vLLM project) Format: compressed tensors (auto detected by vLLM) Scheme: W4A16 ASYM (4 bit asymmetric weights, group size 128, 16 bit activations) Algorithm: AWQ (Activation aware Weight Quantization) Calibration: 128 French samples × 192 tokens from wikimedia/wikipedia (20231101.fr) lm head left in fp16 (preserves output quality) Note: v proj → o proj smoothing is skipped on all 40 layers due to GQA shape mismatch (32 query heads / 8 KV heads). This is standard AWQ behaviour for GQA models — quality impact is marginal since the input layernorm → q/k/v smoothing remains active. Usage with vLLM License Apache 2.0, inherited from Mistral Nemo and the abliterated base.
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy