Ornith-1.0-9B-Engineering-LoRA

Ornith-1.0-9B-Engineering-LoRA

LoRA adapter fine-tuned from ornith-ai/Ornith-1.0-9B for engineering QA style responses.

Lineage

ornith-ai/Ornith-1.0-9B + Epoch 2 QLoRA adapter (checkpoint-1072) -> Ornith-1.0-9B-Engineering-LoRA

The original GGUF checkpoint was not used as the training source. Training used the trainable Hugging Face Transformers checkpoint ornith-ai/Ornith-1.0-9B.

Training

• Training examples: 17,133
• Frozen evaluation examples: 902
• Method: QLoRA
• LoRA rank/alpha/dropout: 32 / 64 / 0.05
• Quantization during training: 4-bit NF4, double quantization, bfloat16 compute
• Sequence length: 1024
• Epochs trained: 3
• Selected checkpoint: Epoch 2 (checkpoint-1072)
• Selection reason: Epoch 2 had the best held-out token F1 and lowest validation loss.

Validation loss:

Frozen Evaluation

All results below use the same frozen 902-example evaluation set, deterministic decoding, and the same evaluator.

Epoch 2 vs Base: 749 improved, 134 regressed, 19 unchanged by token F1.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

base_model = "ornith-ai/Ornith-1.0-9B"
adapter = "Sara121/Ornith-1.0-9B-Engineering-LoRA"

tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    trust_remote_code=True,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)
model = PeftModel.from_pretrained(model, adapter)
model.eval()

Use the Ornith/Qwen chat template from the tokenizer. Ornith responses may include <think>...</think> reasoning before the final answer.

Limitations

This adapter was trained on a domain-specific engineering QA dataset. It should be validated for each deployment domain. The frozen evaluator reports exact match, normalized exact match, and token F1; it does not prove regulatory correctness or eliminate hallucination risk.