OraQwen3-Omni-20B-A3B-Instruct-GGUF

OraQwen3-Omni-20B-A3B-Instruct-GGUF

This repository contains GGUF builds of OraQwen3-Omni-20B-A3B-Instruct developed by Ora Computing. These are llama.cpp-compatible quantizations of oracomputing/OraQwen3-Omni-20B-A3B-Instruct, which is itself a compressed and fine-tuned derivative of the original Qwen/Qwen3-Omni-30B-A3B-Instruct model.

Modalities: text + image + audio input -> text output. This GGUF contains only the Qwen3-Omni thinker (understanding + text generation) and its multimodal input encoders. The talker / code2wav speech-generation path is not included and is not supported by mainline llama.cpp, so there is no speech (audio) output.

---

Model Overview

Model name: OraQwen3-Omni-20B-A3B-Instruct-GGUF
Source model (immediate parent): oracomputing/OraQwen3-Omni-20B-A3B-Instruct (these are GGUF quantizations of it)
Original upstream model: Qwen/Qwen3-Omni-30B-A3B-Instruct
Derived size: ~20 billion total parameters / ~3 billion active parameters (MoE; compressed from the upstream model's ~30 billion total)
Quantization: GGUF Q4_K_M and Q3_K_M, produced with llama.cpp using an importance matrix (i-matrix) computed on Ora's calibration mixture. The thinker backbone is quantized; the vision and audio encoders (the mmproj) are kept at F16. Use Q4_K_M for higher quality or Q3_K_M for a smaller/faster footprint - both share the same mmproj.
Purpose: Evaluation/test-use only; optimized for local/offline inference and internal benchmarking.
License: See LICENSE (Custom Model License Agreement)

---

Files in this repo

---

IMPORTANT: requires a patched llama.cpp

Mainline llama.cpp cannot currently load the full multimodal Qwen3-Omni model (upstream issue ggml-org/llama.cpp#20899) because of how Qwen3-VL/Omni's deepstack vision feature interacts with the audio encoder. The GGUF files here already carry the corrected n_deepstack_layers metadata, but you still need a small two-line source patch to tools/mtmd/mtmd.cpp plus a rebuild.

Everything you need is in llamacpp-patch/:

# from your llama.cpp checkout (commit b9840 recommended)
/path/to/llamacpp-patch/apply_and_build.sh "$PWD"

See llamacpp-patch/README.md for the matching commit, manual edits, and revert instructions.

---

Usage (after patching llama.cpp)

export THINKER=/path/to/OraQwen3-Omni-20B-A3B-Instruct-thinker-Q4_K_M.gguf
export MMPROJ=/path/to/OraQwen3-Omni-20B-A3B-Instruct-mmproj-F16.gguf

Interactive multimodal chat (then use /image <path> or /audio <path>):

./build/bin/llama-mtmd-cli -m "$THINKER" --mmproj "$MMPROJ" -ngl 99

Single-shot image:

./build/bin/llama-mtmd-cli -m "$THINKER" --mmproj "$MMPROJ" -ngl 99 \
  --image /path/to/image.jpg -p "Describe this image in one sentence."

Single-shot audio (understanding / transcription):

./build/bin/llama-mtmd-cli -m "$THINKER" --mmproj "$MMPROJ" -ngl 99 \
  --audio /path/to/audio.mp3 -p "What is said in this audio? Summarize briefly."

OpenAI-compatible server (Web UI at http://localhost:8080):

./build/bin/llama-server -m "$THINKER" --mmproj "$MMPROJ" -ngl 99 \
  --alias oraqwen3-omni --host 0.0.0.0 --port 8080

Text-only inference also works with the standard llama-cli (no mmproj needed).

---

Intended Use & Restrictions

Permitted use

• Internal testing, benchmarking, and evaluation of the model by the named Licensee.
• Exploration of model behaviours, prompt engineering, and non-production prototypes.
• Evaluation across supported input modalities: text, audio, image, and video input; text output (no speech output in this GGUF build).

Prohibited use

• Deployment in a production or commercial service, publicly-facing API, resale, or redistribution.
• Fine-tuning or creating derivative models for production use without a separate agreement.
• Disclosure or sharing of the model (or its weights) to third parties beyond the named Licensee.

Out-of-scope use

• Use of the model in regulated or safety-critical contexts (unless separately permitted).
• Any use that violates the Apache License, Version 2.0 under which the upstream model is distributed.

---

Compression, Fine-Tuning & Quantization

• Parameter count compression: Total parameter count reduced from 30 billion to 20 billion by compressing MLP/expert layers; the MoE active-parameter count (3B) is unchanged.
• Fine-tuning: The compressed model was subsequently fine-tuned on proprietary data curated by Ora Computing.
• GGUF quantization: The fine-tuned thinker was converted to GGUF and quantized to Q4_K_M with llama.cpp, guided by an importance matrix (i-matrix) computed on Ora's calibration mixture, to reduce the memory footprint and accelerate local inference. The multimodal input encoders (mmproj) are retained at F16.

---

Limitations & Risks

• Compressed and quantized models may not replicate the full behaviour of the base model under all prompt categories, particularly domain-specific or rare inputs.
• Audio support in llama.cpp is marked experimental upstream and may have reduced quality.
• The model is provided as-is for testing only and is not certified for production use.
• Users should validate outputs carefully and monitor for bias or unintended behaviours.

---

Upstream Attribution

This model is derived from the Qwen3-Omni-30B-A3B-Instruct model released by Alibaba Cloud under the Apache License, Version 2.0.

"Copyright 2025 Alibaba Cloud. Licensed under the Apache License, Version 2.0."

For full terms, see: https://github.com/QwenLM/Qwen3-Omni/blob/main/LICENSE Apache License, Version 2.0: https://www.apache.org/licenses/LICENSE-2.0

---

Contact & Support

For licensing inquiries or to request extended evaluation rights, please contact: [email protected]

---

Repository and model access are regulated. Do not redistribute or share without explicit written permission from Ora Computing.