symbolic-math-qwen2p5-1p5b-lora

symbolic-math-qwen2p5-1p5b-lora

LoRA adapter (r=8) for Qwen2.5-Math-1.5B, fine-tuned on math word problems. For a standalone model that loads without PEFT, use the merged version: Kevinmastascusa/qwen2p5-math-1p5b-merged. Training, merge and evaluation code is in kevinmastascusa/symbolic-math-ai.

Model details

• Developed by: Kevinmastascusa
• Model type: LoRA adapter for a causal LM
• Language: English (math reasoning)
• License: Apache 2.0, inherited from the base model
• Fine-tuned from: Qwen/Qwen2.5-Math-1.5B

Uses

• Direct use: attach this adapter to the base model to solve math word problems, simple algebra and numeric reasoning.
• Downstream use: further fine-tune it for domain-specific math tasks or curriculum data.

Out-of-scope use

• Safety-critical decision making
• Factual reasoning outside math
• Formal proof generation

How to get started

Use the tokenizer from this adapter repo:

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

adapter = "Kevinmastascusa/symbolic-math-qwen2p5-1p5b-lora"
base = "Qwen/Qwen2.5-Math-1.5B"

tok = AutoTokenizer.from_pretrained(adapter, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(base, trust_remote_code=True)

# Attach LoRA
model = PeftModel.from_pretrained(model, adapter)

prompt = "Question: A shop sells pens at 3 for $2. How much do 12 pens cost?\nAnswer:"
inputs = tok(prompt, return_tensors="pt")
gen = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tok.decode(gen[0], skip_special_tokens=True))

If you see vocab-size mismatch errors, resize the base embeddings to the adapter's tokenizer first:

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

adapter = "Kevinmastascusa/symbolic-math-qwen2p5-1p5b-lora"
base = "Qwen/Qwen2.5-Math-1.5B"

tok = AutoTokenizer.from_pretrained(adapter, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
    base, trust_remote_code=True, ignore_mismatched_sizes=True
)
base_model.resize_token_embeddings(len(tok))
model = PeftModel.from_pretrained(base_model, adapter)

To merge and save a standalone model:

merged = model.merge_and_unload()  # converts to a regular HF model
merged.save_pretrained("qwen2p5-math-1p5b-merged")
tok.save_pretrained("qwen2p5-math-1p5b-merged")

Training details

• Method: LoRA on a 4-bit NF4 quantized base with double quantization (QLoRA style), gradient checkpointing and mixed precision
• LoRA config: r=8, alpha=16, dropout=0.05 on the q, k, v, o, gate, up and down projections
• Data: up to 200 examples each from GSM8K, MathQA, SVAMP and MATH-500, split 80/20 into 640 training and 160 validation examples
• Schedule: 2 epochs (320 optimizer steps), batch size 1 with 4 gradient-accumulation steps, peak learning rate 5e-5 with 50 warmup steps and linear decay, weight decay 0.01, max length 256
• Kept checkpoint: lowest validation loss, at step 200 (loss 0.727, perplexity 2.07); this is the adapter at the root of the repo

Results

Exact match on the project's test splits, with greedy decoding and up to 64 new tokens. The scorer takes the last number in the model's output and compares it with the last number in the gold answer.

• MathQA is not a real score yet. Its gold answers are option letters (a to e), and the scorer compares them with the last number in the output, so they can never match. MathQA needs an option-aware scorer before it can be reported. Counting MathQA's forced zeros, the scorer reports 29.4% over all 700 problems.
• Each set uses the first problems of its test split (MATH-500: the project's 100-problem test split).
• There is no base-model baseline under the same scorer yet, so these numbers show where the model stands, not how much fine-tuning added.
• Script: evaluate_model.py. Raw results: eval_all.json.

Bias, risks and limitations

• Math problems can be ambiguous; outputs may include reasoning lapses or formatting inconsistencies.
• Final-answer formatting affects scoring: the scorer reads the last number in the output.
• Small training set (640 examples) and small test sets (100 to 200 problems each).
• Not robust to adversarial prompts or non-math tasks.
• Potential data biases from the source datasets.

Recommendations

• Use deterministic decoding for answer extraction (no sampling).
• Post-process outputs to extract final numeric answers reliably.

Citation

If you use this adapter, please cite the base model and PEFT:

@article{yang2024qwen25mathtechnicalreportmathematical,
  title={Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement},
  author={An Yang and Beichen Zhang and Binyuan Hui and Bofei Gao and Bowen Yu and Chengpeng Li and Dayiheng Liu and Jianhong Tu and Jingren Zhou and Junyang Lin and Keming Lu and Mingfeng Xue and Runji Lin and Tianyu Liu and Xingzhang Ren and Zhenru Zhang},
  journal={arXiv preprint arXiv:2409.12122},
  year={2024}
}

Framework versions

• PEFT 0.17.0
• Transformers >= 4.41
• PyTorch >= 2.1