Qwen3-4B-ORA-W3

<div align="center"> <img src="assets/ora-logo-extended-white.svg" alt="Ora" width="520"/> <br/> <a href="https://www.oracomputing.com/en"><b>Website</b></a> &nbsp;·&nbsp; <a href="https://www.oracomputing.com/en/blog/ora-qat"><b>Blog</b></a> &nbsp;·&nbsp; <a href="https://www.oracomputing.com/en/contact"><b>Contact</b></a> </div>

Qwen3-4B-ORA-W3

[!IMPORTANT] ~3.7× smaller than the original 16-bit Qwen3-4B, with 96.5% accuracy retention.

3-bit weight-only quantization for Qwen/Qwen3-4B using our propietary Qauntization-Aware-Training pipeline, more information in the dedicated post.

Serve with vLLM ≥ 0.25.0 (Humming WNA16) to keep weights packed. Transformers also works if you pin compressed-tensors>=0.18 but it decompresses the 3-bit weights to bf16 in memory.

Benchmarks

Scores versus the original 16-bit Qwen/Qwen3-4B. Higher is better.

Note: The same setup was applied to all models, so scores in this table are comparable.

<!-- Optional chart: drop a PNG at assets/benchmarks.png and uncomment the next line.  -->

Usage

Serve with vLLM

Packed 3-bit inference. This is the path that keeps the size win.

pip install "vllm>=0.25.0"

vllm serve oracomputing/Qwen3-4B-ORA-W3

The server speaks the OpenAI chat API on http://localhost:8000:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "oracomputing/Qwen3-4B-ORA-W3",
    "messages": [{"role": "user", "content": "Hello"}],
    "temperature": 0.6,
    "top_p": 0.95,
    "top_k": 20,
    "max_tokens": 2048
  }'

Python:

from vllm import LLM, SamplingParams

llm = LLM(model="oracomputing/Qwen3-4B-ORA-W3")
params = SamplingParams(
    temperature=0.6,
    top_p=0.95,
    top_k=20,
    min_p=0.0,
    max_tokens=2048,
)
print(llm.generate(["Hello"], params)[0].outputs[0].text)

If Humming fails with failed to open libnvrtc-builtins.so.13.0, point LD_LIBRARY_PATH at your CUDA 13 NVRTC libs (for example .../site-packages/nvidia/cu13/lib from the PyTorch/NVIDIA wheels).

Transformers (decompresses to bf16)

Pin compressed-tensors ≥ 0.18 so the dense 3-bit pack unpacks correctly. Older 0.17.x will mis-decode these weights. Transformers loads the model as bf16 (not packed 3-bit inference).

pip install "transformers>=4.51" "compressed-tensors>=0.18"

from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.utils.quantization_config import CompressedTensorsConfig

model_id = "oracomputing/Qwen3-4B-ORA-W3"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype="bfloat16",
    device_map="auto",
    quantization_config=CompressedTensorsConfig(run_compressed=False),
)

messages = [{"role": "user", "content": "Hello"}]
text = tok.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,
)
ids = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=256, temperature=0.6, top_p=0.95, top_k=20)
print(tok.decode(out[0], skip_special_tokens=True))

Best Practices

Sampling defaults are the same as Qwen/Qwen3-4B. They are already stored in this checkpoint’s generation_config.json for thinking mode.

1. Sampling parameters

   • Thinking (enable_thinking=True): temperature=0.6, top_p=0.95, top_k=20, min_p=0. Do not use greedy decoding — it degrades quality and can loop forever. These are the defaults in generation_config.json.
   • Non-thinking (enable_thinking=False): temperature=0.7, top_p=0.8, top_k=20, min_p=0.
   • If you hit endless repetition, raise presence_penalty between 0 and 2 (1.5 is a common starting point). Higher values can mix languages and slightly hurt quality.

2. Output length

   • Use up to 32,768 new tokens for most queries.
   • For hard math / coding contest problems, allow up to 81,920.

3. Switching thinking on or off

   Pass enable_thinking through the chat template (Transformers and vLLM both honor this):

   prompt = tokenizer.apply_chat_template(
       messages,
       tokenize=False,
       add_generation_prompt=True,
       enable_thinking=True,   # or False
   )
   

4. Prompt format when you are benchmarking

   • Math: append Please reason step by step, and put your final answer within \boxed{}.
   • Multiple choice: ask the model to put only the letter in JSON, e.g. "answer": "C".

5. Multi-turn history Keep only the final answer in conversation history — drop the <think>...</think> block from earlier turns.