Ornith-1.0-9B-AWQ-INT4

Ornith-1.0-9B-AWQ-INT4

AWQ INT4 (pack-quantized) conversion of Ornith-1.0-9B, optimized for vLLM AWQ kernel inference.

Details

Compatibility

• ✅ vLLM (AWQ kernel, pack-quantized format)
• ✅ SGLang
• ✅ Transformers (with compressed-tensors)

Usage

from vllm import LLM, SamplingParams

llm = LLM(model=spele1100/Ornith-1.0-9B-AWQ-INT4, quantization=awq)
sampling = SamplingParams(temperature=0.6, top_p=0.95, top_k=20)
output = llm.generate([Hello!], sampling)
print(output[0].outputs[0].text)

Quantization Script

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier
from transformers import AutoTokenizer
from datasets import load_dataset

model = ornith-ai/Ornith-1.0-9B
tok = AutoTokenizer.from_pretrained(model, trust_remote_code=True)
ds = load_dataset(HuggingFaceH4/ultrachat_200k, split=train_sft).select(range(512))

recipe = GPTQModifier(
    targets=Linear,
    scheme=W4A16,
    ignore=[model.visual.*],
)

oneshot(
    model=model,
    recipe=recipe,
    output_dir=./Ornith-1.0-9B-AWQ-INT4,
    tokenizer=tok,
    dataset=ds,
    max_seq_length=512,
    num_calibration_samples=128,
)

Hardware

Quantized on a single NVIDIA A100-80GB-SXM4 via RunPod.

License

MIT — inherits from Ornith-1.0-9B.

Acknowledgements

• Ornith-1.0 by Deep Reinforce AI
• llmcompressor by vLLM Project
• cyankiwi/Ornith-1.0-9B-AWQ-INT4 for reference