Holo-3.1-35B-A3B — GGUF Quantizations
GGUF quantizations of Hcompany/Holo-3.1-35B-A3B for use with llama.cpp.
About the Model
- Architecture: Qwen3.5 MoE (35B total, ~3B active parameters)
- Type: Vision-Language Model (VLM) for computer use agents
- Context: 262,144 tokens
- License: Apache 2.0
Note: BF16 GGUF and imatrix sourced from Hcompany/Holo-3.1-35B-A3B-GGUF.
Available Files
| Filename | Quant | Bits | Size (est.) |
|---|---|---|---|
Holo-3.1-35B-A3B-Q2_K.gguf | Q2_K | 2 | ~12 GB |
Holo-3.1-35B-A3B-Q3_K_S.gguf | Q3_K_S | 3 | ~15 GB |
Holo-3.1-35B-A3B-Q3_K_M.gguf | Q3_K_M | 3 | ~17 GB |
Holo-3.1-35B-A3B-Q3_K_L.gguf | Q3_K_L | 3 | ~18 GB |
Holo-3.1-35B-A3B-IQ3_XXS.gguf | IQ3_XXS | 3 | ~14 GB |
Holo-3.1-35B-A3B-IQ3_S.gguf | IQ3_S | 3 | ~15 GB |
Holo-3.1-35B-A3B-Q4_K_S.gguf | Q4_K_S | 4 | ~20 GB |
Holo-3.1-35B-A3B-Q4_K_M.gguf | Q4_K_M | 4 | ~22 GB |
Holo-3.1-35B-A3B-Q5_K_S.gguf | Q5_K_S | 5 | ~24 GB |
Holo-3.1-35B-A3B-Q5_K_M.gguf | Q5_K_M | 5 | ~26 GB |
Holo-3.1-35B-A3B-Q6_K.gguf | Q6_K | 6 | ~30 GB |
Holo-3.1-35B-A3B-Q8_0.gguf | Q8_0 | 8 | ~39 GB |
mmproj-Holo-3.1-35B-A3B-F16.gguf | F16 | 16 | ~0.9 GB |
BF16 GGUF (69.4 GB) available at source repo.
Usage
# Text + Vision inference
huggingface-cli download barozp/Holo-3.1-35B-A3B-GGUF Holo-3.1-35B-A3B-Q4_K_M.gguf --local-dir ./
huggingface-cli download barozp/Holo-3.1-35B-A3B-GGUF mmproj-Holo-3.1-35B-A3B-F16.gguf --local-dir ./
llama-cli \
-m Holo-3.1-35B-A3B-Q4_K_M.gguf \
--mmproj mmproj-Holo-3.1-35B-A3B-F16.gguf \
--jinja \
-ngl 99 \
-c 32768 \
--image screenshot.png
Quantization Details
- BF16 GGUF source: Hcompany/Holo-3.1-35B-A3B-GGUF
- imatrix: Pre-computed, sourced from BF16 repo (192 MB)
- Tool: llama.cpp
llama-quantize - Quantized by: barozp