Qwen3.5-9B-GGUF

<div align="center"> <img src="assets/ora-logo-extended-white.svg" alt="Ora" width="520"/> <br/> <a href="https://www.oracomputing.com/en"><b>Website</b></a> &nbsp;·&nbsp; <a href="https://www.oracomputing.com/en/blog"><b>Blog</b></a> &nbsp;·&nbsp; <a href="https://www.oracomputing.com/en/contact"><b>Contact</b></a> </div>

Qwen3.5-9B-GGUF (OraQuant)

This repository contains GGUF builds of Qwen3.5-9B, quantized by Ora Computing with OraQuant (OQ) - Ora Computing's proprietary calibrated quantization. These are llama.cpp-compatible quantizations of Qwen/Qwen3.5-9B; the underlying weights are unchanged Qwen3.5-9B weights at reduced precision.

Text only. Qwen/Qwen3.5-9B is a multimodal model; these GGUFs contain only the language model (text input -> text output). The vision/video input encoders are not included.

---

Model Overview

• Model name: Qwen3.5-9B-GGUF (OraQuant)
• Base model: Qwen/Qwen3.5-9B (Apache-2.0, Alibaba Cloud) - these are GGUF quantizations of it
• Parameters: ~9 billion (unchanged from the base model)
• Quantization: OraQuant (OQ) mixed-precision GGUFs produced by Ora Computing, provided in three footprints - OQ-Q4_K_M (higher quality), OQ-Q3_K_M (smaller/faster), and OQ-IQ2_XXS (smallest, mixed i-/k-quant).
• Not fine-tuned, not parameter-reduced: the model architecture and parameter count are identical to the base model; only the weight precision is reduced.
• Purpose: Evaluation/test-use only; optimized for local/offline inference and internal benchmarking.
• License: See LICENSE (Custom Model License Agreement).

---

Files in this repo

---

Usage

These GGUFs load with stock upstream llama.cpp (no patch required); use a build with Qwen3.5 support.

export MODEL=/path/to/Qwen3.5-9B-OQ-Q4_K_M.gguf   # or the Q3_K_M / IQ2_XXS file

Interactive chat:

./build/bin/llama-cli -m "$MODEL" -ngl 99

Single-shot completion (-st runs one turn then exits):

./build/bin/llama-cli -m "$MODEL" -ngl 99 -st -p "Explain the Chudnovsky algorithm in two sentences."

OpenAI-compatible server (Web UI at http://localhost:8080):

./build/bin/llama-server -m "$MODEL" -ngl 99 \
  --alias qwen3.5-9b --host 0.0.0.0 --port 8080

Qwen3.5 is a reasoning model; the chat template and thinking behaviour are carried in the GGUF. For best results use the upstream-recommended sampling settings, e.g. --temp 0.6 --top-p 0.95 --top-k 20 --presence-penalty 1.0. Long thinking phases are normal on hard prompts, especially for the smallest (OQ-IQ2_XXS) footprint.

---

Intended Use & Restrictions

Permitted use

• Internal testing, benchmarking, and evaluation of the model by the named Licensee.
• Exploration of model behaviours, prompt engineering, and non-production prototypes.

Prohibited use

• Deployment in a production or commercial service, publicly-facing API, resale, or redistribution.
• Fine-tuning or creating derivative models for production use without a separate agreement.
• Reverse-engineering the quantization/calibration used to produce these files.
• Disclosure or sharing of the model (or its weights) to third parties beyond the named Licensee.

Out-of-scope use

• Use in regulated or safety-critical contexts (unless separately permitted).
• Any use that violates the Apache License, Version 2.0 under which the upstream model is distributed.

---

Quantization

• Method: OraQuant (OQ), Ora Computing's proprietary calibrated quantization. The released files are mixed-precision GGUFs (K-quants; the OQ-IQ2_XXS file additionally uses i-quants).
• No fine-tuning: the weights are the original Qwen/Qwen3.5-9B weights; no additional training was performed.
• No parameter-count change: the architecture and ~9B parameter count are unchanged; only weight precision is reduced.
• Footprints: OQ-Q4_K_M for higher quality, OQ-Q3_K_M for a smaller/faster footprint, OQ-IQ2_XXS for the smallest footprint (expect visible quality loss on the hardest long-reasoning tasks at this size).

---

Limitations & Risks

• Quantized models may not replicate the full behaviour of the base model under all prompt categories, particularly domain-specific or rare inputs.
• The model is provided as-is for testing only and is not certified for production use.
• Users should validate outputs carefully and monitor for bias or unintended behaviours.

---

Upstream Attribution

This model is derived from the Qwen3.5-9B model released by Alibaba Cloud under the Apache License, Version 2.0.

"Copyright 2025 Alibaba Cloud. Licensed under the Apache License, Version 2.0."

For full terms, see: https://huggingface.co/Qwen/Qwen3.5-9B/blob/main/LICENSE Apache License, Version 2.0: https://www.apache.org/licenses/LICENSE-2.0

---

Contact & Support

For licensing inquiries or to request extended evaluation rights, please contact: [email protected]

---

Repository and model access are regulated. Do not redistribute or share without explicit written permission from Ora Computing.