siba-meristem-1.0

<div align="center">

SIBA Meristem 1.0

Multilingual Large Language Model for Caucasian and Eurasian Languages

Azerbaijani | Russian | English | Turkish

SIBA's first open model: a multilingual adapter aimed at Azerbaijani and the Caucasus, for research use.

Built with Qwen

</div>

---

Update, 25 September 2026: licence corrected, first benchmark, version 1.0.1

Licence. Meristem 1.0 is a LoRA adapter on a base model released under the Qwen Research License (research use only, non-commercial). An adapter cannot carry a looser licence than the model it modifies, and part of its training data (tatsu-lab/alpaca) is CC BY-NC 4.0, so this repository was wrongly labelled Apache 2.0. It is now labelled correctly and a copy of the Qwen Research License is in LICENSE.

First benchmark. Measured on azbench, 2,148 zero-shot Azerbaijani multiple-choice questions, exact-letter scoring, run locally in Q4_K_M:

Meristem 1.0 scores below its own base. It was released at step 500 of a planned 31,461 and learned to favour the letter "C" (82 % of its answers on the reading set).

Version 1.0.1 (in v1.0.1/) continues the same LoRA for 20 optimizer steps on letter-balanced Azerbaijani cloze questions built from Azerbaijani Wikipedia (CC BY-SA), loss on the answer letter only, none of it from the azbench test sets. On the first 300 reading questions (all three models in Q4_K_M, temperature 0, the base model's default system prompt):

1.0.1 removes the "C" bias and recovers most of the loss, but is still below the base model. It is a repair, not an improvement over its base model. The next Meristem will be built on a commercially licensed base and only released if it beats that base on azbench.

Full azbench run for 1.0.1, 28 September 2026. All 2,148 questions on an RTX 3090, Q4_K_M through Ollama, temperature 0, each model with its own Ollama template. The Ollama tag latest is 1.0.1 (same digest, 80deca0d15a4). This setup differs from the tables above, which used the base model's default system prompt for every model, so compare rows within a table and not across tables.

On the full set 1.0.1 is 6.3 points below its base on reading, 3.1 points above it on exam questions (within noise at 548 questions) and 1.7 above on school questions, where both models are below the 42.7 % of always answering D. On average it is still just below the base model. Raw counts: github.com/aghasalim/azbench.

---

Highlights

• 167,785 multilingual instruction pairs (~32.5M tokens) spanning 4 languages
• QLoRA fine-tuned by SIBA with Unsloth for maximum efficiency
• Training loss reduced from 2.49 to 1.26 (49.4% reduction) over 500 gradient steps
• Available in GGUF Q4_K_M (1.8 GB) for efficient local inference
• Available as LoRA adapters for custom fine-tuning and research
• Runs on consumer hardware -- 4GB+ VRAM sufficient for inference

---

Training Details

Architecture and Method

Training Loss Curve

Loss reduction: 2.491 to 1.262 (49.4% improvement)

Dataset Composition

---

Quick Start

Option 1: Ollama (Easiest)

# Install Ollama from https://ollama.ai, then:
ollama run aghasalim/siba-meristem-1.0

Option 2: llama.cpp / LM Studio

Download Qwen2.5-3B-Instruct.Q4_K_M.gguf from this repository and load it in any GGUF-compatible runtime:

# llama.cpp example
./llama-cli -m Qwen2.5-3B-Instruct.Q4_K_M.gguf \
  -p "<|im_start|>user\nSalam! Azerbaycan haqqinda melumat ver.<|im_end|>\n<|im_start|>assistant\n" \
  -n 256 --temp 0.7 --top-p 0.9

Option 3: Transformers + PEFT (Python)

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

# Load base model and LoRA adapter
base_model_id = "Qwen/Qwen2.5-3B-Instruct"
adapter_id = "aghasalim/siba-meristem-1.0"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)
model = PeftModel.from_pretrained(model, adapter_id)

# Chat in Azerbaijani
messages = [
    {"role": "system", "content": "Sen SIBA Meristem 1.0, coxdilli AI komekcisin."},
    {"role": "user", "content": "Baki seheri haqqinda qisa melumat ver."}
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        do_sample=True
    )

response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
print(response)

Option 4: Unsloth (Fastest Inference)

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="aghasalim/siba-meristem-1.0",
    max_seq_length=8192,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

messages = [
    {"role": "user", "content": "Hello! Tell me about yourself."}
]

inputs = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
).to("cuda")

outputs = model.generate(input_ids=inputs, max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

Repository Contents

---

Multilingual Examples

<details> <summary><b>Azerbaijani Example</b></summary>

User: Azerbaijan metbexinin en meshur yemekleri hansilardir?

SIBA: Azerbaijan metbexi zengin ve muxtelifdir. En meshur yemekler bunlardir:

1. Plov (As) -- Azerbaijan metbexinin taci sayilir
2. Dolma -- Uzum yarpagina bukulmus et-duyu qarisigi
3. Qutab -- Nazik xemirden hazirlanan yasti borek
4. Piti -- Susa bolgesinin enenevi sorbasi
5. Lulekabab -- Manqalda bisirilen et kabab

</details> <details> <summary><b>Russian Example</b></summary>

User: Explain the difference between machine learning and deep learning.

SIBA: Machine learning (ML) and deep learning (DL) are related but distinct fields:

Machine learning is a broad area of AI where algorithms learn from data...
Deep learning is a subset of ML that uses neural networks with many layers...

</details> <details> <summary><b>Turkish Example</b></summary>

User: Turkiye'nin en guzel turistik yerleri nerelerdir?

SIBA: Turkiye, zengin tarihi ve dogal guzellikleriyle dolu bir ulkedir:

1. Istanbul -- Ayasofya, Topkapi Sarayi, Kapalicars
2. Kapadokya -- Peri bacalari ve sicak hava balonlari
3. Pamukkale -- Beyaz travertenler ve antik Hierapolis

</details> <details> <summary><b>English Example</b></summary>

User: What makes the Caucasus region culturally unique?

SIBA: The Caucasus region is one of the most culturally diverse areas on Earth:

1. Linguistic Diversity -- Over 50 distinct languages spoken across a relatively small area
2. Ancient Heritage -- Civilizations dating back to the Bronze Age
3. Cultural Crossroads -- Intersection of European, Asian, and Middle Eastern influences

</details>

---

Limitations and Known Issues

• Early checkpoint: This is from step 500 of a planned 31,461-step training run. That run was not continued; see the update at the top.
• Azerbaijani understanding: On azbench, Meristem 1.0 scores below its base model (32.9 % against 39.5 %), and 1.0.1 is still below it on the reading set. Do not assume it understands Azerbaijani better than its base model.
• Factual accuracy: Like all LLMs, the model may generate plausible-sounding but incorrect information. Always verify critical facts.
• Context window: The model supports up to 8,192 tokens of context.
• Not safety-tuned: This model has not undergone RLHF or constitutional AI safety training beyond what is inherited from its base model.

---

License

Built with Qwen: SIBA Meristem 1.0 is fine-tuned by SIBA from Qwen2.5-3B-Instruct and is governed by the Qwen Research License Agreement of that model: research and evaluation use only, not commercial use. A copy is in LICENSE, and the required attribution is in NOTICE. Part of the training data (tatsu-lab/alpaca) is also CC BY-NC 4.0.

Until 25 September 2026 this repository was labelled Apache 2.0. That label was wrong and has been corrected; it did not change the terms of the base model, which always applied.

---

Links

---

Acknowledgments

• Qwen Team (Alibaba Cloud) for the Qwen2.5 base model
• Unsloth for efficient 4-bit QLoRA fine-tuning on consumer GPUs
• Hugging Face for model hosting and ecosystem
• Ollama for accessible local LLM deployment

---

Contact

Developer: Aghasalim Mustafazada Hugging Face: @aghasalim Ollama: @aghasalim

---

<div align="center"> <i>SIBA Meristem -- Growing intelligence, one language at a time.</i> </div>