Ornith-1.0-35B-int4-AutoRound

Ornith-1.0-35B-A3B-int4-AutoRound

A 4-bit AutoRound quantized version of deepreinforce-ai/Ornith-1.0-35B, optimized for efficient inference on consumer hardware including the NVIDIA DGX Spark (GB10).

Model Details

Architecture

• Type: Qwen3.5-MoE with Vision (multimodal)
• Hidden Size: 2,048
• Num Layers: 40
• Num Experts: 256
• Experts Per Token: 8
• Attention Heads: 16 (full attention every 4th layer, linear attention otherwise)
• Vocab Size: 248,320

Quantization Details

This model was quantized using AutoRound, an advanced quantization technique from Intel that uses signed gradient descent to jointly optimize weight rounding and clipping ranges.

Key Settings

--batch_size 8
--iters 1000
--nsamples 512
--seqlen 2048
--dataset opencode-instruct
--group_size 128

Quantization Quality

The model retains strong quality with all layers meeting minimum thresholds. Layers 22-39 show slightly elevated sensitivity (cosine similarity 0.986-0.990) which is typical for deeper MoE layers.

Note: Shared expert gates are preserved in FP16 to maintain MoE routing accuracy.

Usage with Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "whpthomas/Ornith-1.0-35B-int4-AutoRound"

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_path)

# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    device_map="auto",
    torch_dtype="auto",
)

# Generate text
messages = [
    {"role": "user", "content": "Explain quantum computing in simple terms."}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Usage with vLLM

For optimal performance, use vLLM with the recommended serving configuration:

Installation

pip install vllm

Basic Serving

vllm serve whpthomas/Ornith-1.0-35B-int4-AutoRound \
  --served-model-name ornith-1.0-35b \
  --max-model-len 196608 \
  --gpu-memory-utilization 0.55 \
  --load-format auto \
  --attention-backend flashinfer \
  --moe-backend marlin \
  --enable-prefix-caching \
  --enable-chunked-prefill

Recommended Production Configuration

For DGX Spark (GB10) with optimal performance:

vllm serve whpthomas/Ornith-1.0-35B-int4-AutoRound \
  --served-model-name ornith-1.0-35b \
  --max-model-len 196608 \
  --gpu-memory-utilization 0.55 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 8 \
  --optimization-level 3 \
  --performance-mode throughput \
  --load-format instanttensor \
  --attention-backend flashinfer \
  --moe-backend marlin \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --default-chat-template-kwargs '{"preserve_thinking":true}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --generation-config auto \
  --override-generation-config '{"temperature":0.7,"top_p":0.95,"top_k":-1,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

Environment Variables (DGX Spark)

export TORCH_MATMUL_PRECISION=high
export NVIDIA_FORWARD_COMPAT=1
export NVIDIA_DISABLE_REQUIRE=1
export CUDA_DEVICE_MAX_CONNECTIONS=1
export VLLM_MARLIN_USE_ATOMIC_ADD=1
export FLASHINFER_DISABLE_VERSION_CHECK=1

Tool Calling

This model supports tool calling with the Qwen3 coder parser. When using vLLM with --enable-auto-tool-choice --tool-call-parser qwen3_coder, the model can invoke tools and return structured function calls.

Reasoning

The model supports extended thinking with the Qwen3 reasoning parser. Use --reasoning-parser qwen3 and set preserve_thinking: true in chat template kwargs to enable reasoning traces.

Performance Characteristics

Based on testing with the Qwen3.5-MoE architecture on DGX Spark:

Performance varies based on context length, batch size, and hardware configuration.

Citation

If you use this quantized model, please cite both the base model and the quantization tool:

@misc{ornith2025,
  title={Ornith-1.0-35B},
  author={DeepReinforce AI},
  year={2025},
  publisher={HuggingFace},
  url={https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B}
}

@misc{sparkautaround2025,
  title={Spark Auto Round},
  author={whpthomas},
  year={2025},
  publisher={GitHub},
  url={https://github.com/whpthomas/spark-auto-round}
}

Acknowledgments

• Base Model: deepreinforce-ai/Ornith-1.0-35B - The original bf16 model
• Quantization: Intel auto-round - The underlying quantization framework
• Spark Auto Round: spark-auto-round - GB10-optimized quantization wrapper
• vLLM Serving: spark-vllm-docker - Community vLLM Docker configuration for DGX Spark

License

The quantized model inherits the license of the base model. Please refer to the original model card for license details.

Model Type

This is an int4 AutoRound quantized model. It requires:

• A CUDA-capable GPU
• vLLM (recommended) or transformers with auto-round support
• Sufficient GPU memory (~20GB+ depending on context length)