saraiki-qwen-8b-tool-use

Saraiki Qwen3-8B — Instruction / Assistant SFT

A LoRA adapter that turns Qwen3-8B-Base into an instruction-following conversational assistant for Jataki Saraiki (سرائیکی).

This is Stage 2 of a three-stage pipeline:

Qwen/Qwen3-8B-Base
    │
    ├─▶ Stage 1: Continued pretraining (CPT) on Saraiki text
    │   themohal/saraiki-qwen3-8b-cpt
    │
    ├─▶ Stage 2: Instruction/assistant SFT  ◀── this model
    │   themohal/saraiki-qwen-8b-sft
    │
    └─▶ Stage 3: Tool-use / function-calling SFT
        themohal/saraiki-qwen-8b-tool-use

Stage 2 merges the finished Stage 1 CPT adapter into the base weights, then trains a fresh LoRA adapter on top for instruction-following — so this model is Saraiki-fluent (from Stage 1's continued pretraining on Saraiki text) and able to hold a conversation and follow instructions, rather than just completing Saraiki text.

What this model does

Given a Saraiki instruction or question, the model responds conversationally in Saraiki — general assistant-style behavior (answering questions, explaining things, following instructions) grounded in the Saraiki continued-pretraining from Stage 1. It does not call tools/functions — that capability is added on top in Stage 3.

Usage

from unsloth import FastLanguageModel
from peft import PeftModel
from transformers import AutoTokenizer
import torch

BASE_MODEL = "Qwen/Qwen3-8B-Base"
CPT_ADAPTER = "themohal/saraiki-qwen3-8b-cpt"
SFT_ADAPTER = "themohal/saraiki-qwen-8b-sft"

tokenizer = AutoTokenizer.from_pretrained(SFT_ADAPTER)

model, _ = FastLanguageModel.from_pretrained(
    model_name=BASE_MODEL, load_in_4bit=True, dtype=None,
)
model.resize_token_embeddings(len(tokenizer))

# Merge chain: base -> CPT -> SFT
model = PeftModel.from_pretrained(model, CPT_ADAPTER).merge_and_unload()
model = PeftModel.from_pretrained(model, SFT_ADAPTER)

messages = [{"role": "user", "content": "میڈا ناں کیا اے؟"}]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,  # see "Thinking mode" below
)

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Thinking mode

Qwen3 defaults to thinking mode on (<think>...</think> before every response). This model was trained with enable_thinking=False — themohal/saraiki-assistant-sft is plain instruction/response data with no reasoning traces, so training with thinking mode on would just teach the model to emit empty <think>\n\n</think> blocks as noise before every answer. Set enable_thinking=False at inference time to match training. Stage 3 was trained matching this same setting for consistency across the pipeline.

Training data

themohal/saraiki-assistant-sft — Saraiki question/answer conversation pairs, generated and validated through → validate → repair → dedupe pipeline.

This dataset grows daily (roughly once per morning), so this model is retrained continually rather than as a single fixed run — see Continual training below.

Training procedure

• Method: LoRA (r=16, alpha=32, dropout=0.0) on q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj; base model weights frozen throughout.
• Loss: assistant_only_loss=True (TRL) — loss computed only on assistant-generated spans, not on user turns.
• Tokenizer extension: two verified Saraiki-specific Arabic-script characters (ڻ / ݙ) added as dedicated tokens; their embedding rows are initialized from the mean of their original sub-token pieces and re-derived identically every training session (they are frozen base-model weights, not part of any LoRA adapter).
• Max sequence length: 2048.
• Trainer: TRL SFTTrainer, dataset auto-detected and converted to the messages conversational format.

Continual training

Because the training dataset grows daily, this model does not use naive resume_from_checkpoint on every run — that assumes a static dataset, which a growing corpus violates (the train/validation split membership and packed-sequence composition change as rows are added). Instead:

• Every training session pins the dataset to a specific commit SHA and records it in a data_manifest.json alongside each pushed checkpoint.
• On the next run, that SHA is compared against the dataset's current SHA:
  • Unchanged → a true resume_from_checkpoint (same optimizer state, same LR schedule position) — this is what happens on same-day reruns.
  • Changed (the normal case, once per day after the morning data update) → a weights-only continuation: adapter weights are carried forward, and a fresh optimizer/scheduler phase trains over the current, larger dataset.

This means the adapter is best understood as a continually-improving artifact rather than a single frozen release — check the commit history / data_manifest.json in the most recent checkpoint folder for what it was actually trained on.

Intended use & limitations

• Built as a general-purpose Saraiki-language conversational assistant, for a specific, currently-unfilled niche — Jataki Saraiki has essentially no existing LLM tooling. It is not intended to compete with frontier-scale general models on broad reasoning or capability; at 8B parameters with LoRA fine-tuning, its ceiling is "the best Saraiki-speaking assistant available," not general intelligence.
• No tool/function-calling ability — see Stage 3 for that.
• It has not yet been evaluated against a held-out, hand-written Saraiki eval set — treat outputs with appropriate caution until that evaluation exists.

Related repositories

Author

Muhammad Farjad Ali Raza

License

Apache 2.0, inherited from the Qwen3-8B-Base license. Training data is synthetically generated; see the dataset card for generation details.