qwen2.5-0.5b-gsm8k-tr-lora

Qwen2.5-0.5B-Instruct - Turkish Math Reasoning (LoRA)

LoRA adapter for Qwen/Qwen2.5-0.5B-Instruct, fine-tuned on ytu-ce-cosmos/gsm8k_tr for Turkish grade-school math word problems. The model produces a step-by-step Turkish solution followed by the final numeric answer.

This repository contains the PEFT adapter only (rank-16 LoRA); the base model is loaded separately at inference time.

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "Qwen/Qwen2.5-0.5B-Instruct"
ADAPTER = "thealper2/qwen2.5-0.5b-gsm8k-tr-lora"

tokenizer = AutoTokenizer.from_pretrained(ADAPTER)
model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

messages = [
    {"role": "system", "content": 'Sen matematik problemlerini cozen bir asistansin. Problemi adim adim Turkce olarak coz ve sonunda nihai cevabi acikca belirt.'},
    {"role": "user", "content": "Ali'nin 5 elmasi vardir. 3 elma daha alirsa kac elmasi olur?"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)

output = model.generate(**inputs, max_new_tokens=400, do_sample=False)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The system prompt above is the one used during training; keeping it improves output-format consistency.

Training data

Examples are rendered with the official Qwen2.5 chat template as system / user / assistant turns. The reference solution is used verbatim; no reasoning steps were rewritten, simplified or synthesised.

Training procedure

Loss is computed on assistant tokens only: prompt tokens are set to -100 via prompt-prefix masking, so the model is not trained to reproduce the question.

lm_head is excluded from the adapter because Qwen2.5-0.5B ties its input and output embeddings.

Sequence length was set from the measured token distribution under the chat template (mean 265, median 253, p90 399, p95 456, p99 589, max 916). At 640 tokens, 99.48% of examples fit; the remaining over-length examples are dropped rather than truncated, because truncation would remove the concluding sentence that states the final answer.

Evaluation

Greedy decoding (do_sample=False, max_new_tokens=400) on 500 held-out validation examples. Both rows are scored on exactly the same examples with the same answer extractor.

Answer accuracy compares the final numeric answer extracted from the generation against the one extracted from the Turkish reference solution, with tolerance 1e-4. The extractor resolves \boxed{}, explicit Cevap: markers, LaTeX fractions, percentages, and both Turkish and English decimal/thousands conventions.

Perplexity is computed over assistant tokens only. It measures fit to the reference solution style, not mathematical correctness -- a model can lower it by imitating phrasing while still computing wrong answers. Answer accuracy is the metric that speaks to reasoning.

Limitations

• Scope. Trained only on GSM8K-style grade-school word problems. Performance here does not imply general mathematical reasoning: no algebra, geometry, proofs, symbolic manipulation or multi-domain math is covered.
• Model capacity. At 0.5B parameters, multi-step arithmetic remains error-prone. Expect arithmetic slips on longer chains.
• Label noise. Roughly 21% of gsm8k_tr solutions reach a different final answer than the corresponding original English GSM8K item (measured on 4592 uniquely alignable rows). Accuracy is measured against the Turkish reference, which is what the model is trained to reproduce, so the reported number reflects agreement with this dataset rather than with original GSM8K.
• Language. Trained and evaluated in Turkish only.
• No verification. Output is not checked by a calculator or solver. Do not rely on it for consequential calculations.