Ornith-1.0-35B GPTQ-Pro FOEM 4-bit g128 ns256

These models are built and maintained on rented GPU compute. If you want to show some appreciation, a follow on X or a coffee helps keep the releases coming.

This is a GPTQ-Pro 4-bit quantization of deepreinforce-ai/Ornith-1.0-35B.

It is a deployment artifact, not a new fine-tune. The goal is to make the Ornith 35B MoE checkpoint practical for GPTQ-compatible local and homelab serving while preserving the coding-agent oriented behavior of the source model.

The headline validation target for this release is text-generation and agentic coding workload serving with vLLM + GPTQ-Marlin. Vision paths and speculative decoding are not positioned as validated features for this release.

Source And Credits

Source model:

• deepreinforce-ai/Ornith-1.0-35B

Quantization tooling and reference recipe:

• modelcloud/gptqmodel
• groxaxo/GPTQ-Pro
• groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit

Thanks to deepreinforce-ai for the Ornith release, to modelcloud for GPTQModel, and to groxaxo for GPTQ-Pro and the Qwen3.6 GPTQ-Pro recipe family this run was aligned with.

Artifact Summary

The source config contains mtp_num_hidden_layers=1, but the uploaded weight index contains no mtp.* tensors. Treat this checkpoint as non-MTP for speculative decoding unless a follow-up artifact restores and validates real MTP tensors.

Vision configuration is present from the source architecture, but multimodal serving has not been validated for this quantized release. The intended use case is text generation and coding-agent evaluation.

Quantization Recipe

Excluded from quantization by dynamic rules:

• embeddings
• lm_head
• MTP modules
• norms
• vision / visual modules

The calibration data itself is not included in this model repository.

Intended Serving Shape

This checkpoint is intended for advanced users testing text-only vLLM or GPTQ-compatible serving for Qwen/Ornith MoE checkpoints.

A starting vLLM shape for long-context text serving:

vllm serve XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256 \
  --served-model-name ornith-1.0-35b-gptq-pro-foem-4bit-g128-ns256-ctx262k \
  --language-model-only \
  --dtype float16 \
  --quantization gptq_marlin \
  --tensor-parallel-size 1 \
  --max-model-len 262144 \
  --max-num-seqs 1 \
  --kv-cache-dtype fp8_e5m2 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.95 \
  --trust-remote-code

Serving context for the published Smoke24/vLLM measurements:

The Smoke24/vLLM numbers were collected on an internal llm-residency vLLM deployment. The custom image recipe is not published yet, so this card does not present that image as a public reproduction target. The stable serving knobs captured from the run are listed for context.

Treat this as a serving starting point, not a compatibility guarantee for every vLLM release. GPTQ-Marlin, Qwen3.5 MoE handling, FP8 KV cache, and tool parser behavior are loader-version sensitive.

Public vLLM Reproducibility

This artifact has a public reproducibility path on the unmodified upstream vLLM OpenAI image:

• image: docker.io/vllm/vllm-openai:nightly-7a1eb8ac2ec4ea69338c51dc7afd4b15010abfa8
• vLLM version observed in validation: 0.20.1rc1.dev16+g7a1eb8ac2
• GPU class: single RTX 3090 24 GB / Ampere
• --enforce-eager was not used
• no local sleep/wake patch or localhost/*sleepwake* image is required for the validation below

Validated serving shape:

• context: --max-model-len 262144 with --gpu-memory-utilization 0.96
• --language-model-only, --dtype float16, --quantization gptq_marlin
• --kv-cache-dtype fp8_e5m2, --enable-prefix-caching, --max-num-seqs 1
• --max-num-batched-tokens 2096, --max-cudagraph-capture-size 32
• --reasoning-parser qwen3, --tool-call-parser qwen3_xml

The 262k profile is tight on 24 GB GPUs; gpu_memory_utilization=0.95 was short on KV cache in validation, while 0.96 passed.

vLLM RTX 3090 / Ampere Workload Snapshot

The image above reports observed vLLM / Prometheus metrics for the served model label ornith-1.0-35b-gptq-pro-foem-4bit-g128-ns256-ctx262k over the recent workload window that included validation traffic:

These are serving metrics, not a standalone quality benchmark. They describe the observed vLLM GPTQ-Marlin runtime behavior for this deployment shape.

MMLU-Pro 350 Selected Subset

Terminal-Bench 2.0 Smoke24

Terminal-Bench 2.0 Smoke24 is a fixed 24-task coding-agent comparison corpus. It is useful for fast regression and local serving comparison, but it is not a full Terminal-Bench leaderboard submission.

XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256 used the long-context card-validation shape: max_model_len=262144, max_input_tokens=220000, 30 minute task timeout, 32 CPU / 48 GiB sandbox, thinking_token_budget=32768, max_output_tokens=40000, temperature 1.0, top-p 0.95, top-k 20, and preserve_thinking=true.

Smoke24 task list and harness summary:

• benchmarks/terminal-bench-2.0/smoke24_task_list_20260616.md

Validation Status

Completed:

• Source model loaded for quantization.
• Code-oriented calibration mix generated.
• GPTQ-Pro FOEM quantization completed.
• Final checkpoint saved as five safetensors shards.
• Hugging Face repo file list verified after upload.
• vLLM serving metrics collected through Prometheus.
• Terminal-Bench 2.0 Smoke24 validation run completed.

Not yet validated:

• Full Terminal-Bench leaderboard submission.
• Vision or multimodal serving.
• MTP/speculative decoding.
• Broad multi-GPU serving matrix.

Limitations

• Experimental quantization.
• Smoke24 is a small 24-task slice; read one-task differences with caution.
• The calibration mix is code-heavy and was chosen for coding-agent use, not as a general-purpose calibration corpus.
• Treat this release as non-MTP for speculative decoding.
• Vision support is not validated.
• Loader behavior may vary across vLLM, Transformers, GPTQModel, and GPTQ-Marlin versions.

Files

Key files:

• model.safetensors.index.json
• model-00001-of-00005.safetensors through model-00005-of-00005.safetensors
• config.json
• generation_config.json
• processor_config.json
• quantize_config.json
• tokenizer.json
• tokenizer_config.json
• chat_template.jinja

References

• Source model: deepreinforce-ai/Ornith-1.0-35B
• GPTQModel: modelcloud/gptqmodel
• GPTQ-Pro tooling: groxaxo/GPTQ-Pro
• Reference recipe: groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit
• Terminal-Bench: laude-institute/terminal-bench

Individual Project Notice

This repository is an individual research project. It is not affiliated with, sponsored by, or endorsed by any employer or organization.