qwen2p5-math-1p5b-merged

qwen2p5-math-1p5b-merged

Qwen2.5-Math-1.5B fine-tuned with LoRA on math word problems, with the adapter merged into the full weights so it loads like any Transformers model. The unmerged adapter is Kevinmastascusa/symbolic-math-qwen2p5-1p5b-lora. Training, merge and evaluation code is in kevinmastascusa/symbolic-math-ai, whose Streamlit app adds Tree-of-Thoughts search with SymPy-checked final answers on top of this model.

Model summary

• Base: Qwen/Qwen2.5-Math-1.5B
• Type: causal LM (decoder-only), about 1.5B parameters
• Weights: LoRA adapter merged into the base weights (merge_lora_to_full.py)
• Focus: math word problems with a final numeric or short text answer
• Prompt formats:
  • GSM8K and SVAMP: Question: ...\nAnswer:
  • MATH-500: Problem: ...\nSolution:
  • MathQA: Problem: ...\nOptions: ...\nSolution:

Results

Exact match on the project's test splits, with greedy decoding and up to 64 new tokens. The scorer takes the last number in the model's output and compares it with the last number in the gold answer.

• MathQA is not a real score yet. Its gold answers are option letters (a to e), and the scorer compares them with the last number in the output, so they can never match. MathQA needs an option-aware scorer before it can be reported. Counting MathQA's forced zeros, the scorer reports 29.4% over all 700 problems.
• Each set uses the first problems of its test split (MATH-500: the project's 100-problem test split).
• There is no base-model baseline under the same scorer yet, so these numbers show where the model stands, not how much fine-tuning added.
• Scored as the run-2 LoRA adapter on the base model; this repo is that adapter merged into the base weights. Script: evaluate_model.py. Raw results: eval_all.json.

Training

• Method: LoRA on a 4-bit NF4 quantized base with double quantization (QLoRA style), gradient checkpointing and mixed precision
• LoRA: r=8, alpha=16, dropout=0.05 on the q, k, v, o, gate, up and down projections
• Data: up to 200 examples each from GSM8K, MathQA, SVAMP and MATH-500, split 80/20 into 640 training and 160 validation examples
• Schedule: 2 epochs (320 optimizer steps), batch size 1 with 4 gradient-accumulation steps, peak learning rate 5e-5 with 50 warmup steps and linear decay, weight decay 0.01, max length 256
• Kept checkpoint: lowest validation loss, at step 200 (loss 0.727, perplexity 2.07)

Intended use

• Math word problem solving and reasoning research
• Not for high-stakes decisions, formal proofs, or competition-level math

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "Kevinmastascusa/qwen2p5-math-1p5b-merged"
tok = AutoTokenizer.from_pretrained(model_id)
if tok.pad_token is None:
    tok.pad_token = tok.eos_token

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
    device_map="auto" if torch.cuda.is_available() else None,
)

def solve(q, max_new_tokens=64):
    prompt = f"Question: {q}\nAnswer:"
    inputs = tok(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(
            **inputs, max_new_tokens=max_new_tokens,
            do_sample=False, num_beams=1, pad_token_id=tok.eos_token_id
        )
    return tok.decode(out[0], skip_special_tokens=True)

Prompting tips

• GSM8K and SVAMP: Question: ...\nAnswer:
• MATH-500: Problem: ...\nSolution:
• MathQA: Problem: ...\nOptions: ...\nSolution:

Limitations

• Final-answer formatting affects scoring: the scorer reads the last number in the output
• Small training set (640 examples) and small test sets (100 to 200 problems each)
• A 64-token limit can cut off longer solutions
• May hallucinate outside math tasks; use with caution

Hardware and latency

• About 1.5B parameters; a GPU is recommended for interactive use
• Typical decoding: 32 to 64 new tokens

License

Apache 2.0, inherited from the base model (Qwen/Qwen2.5-Math-1.5B).

Citation

If you use this model, please cite the base model:

@article{yang2024qwen25mathtechnicalreportmathematical,
  title={Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement},
  author={An Yang and Beichen Zhang and Binyuan Hui and Bofei Gao and Bowen Yu and Chengpeng Li and Dayiheng Liu and Jianhong Tu and Jingren Zhou and Junyang Lin and Keming Lu and Mingfeng Xue and Runji Lin and Tianyu Liu and Xingzhang Ren and Zhenru Zhang},
  journal={arXiv preprint arXiv:2409.12122},
  year={2024}
}

Acknowledgements

• The Qwen team at Alibaba for Qwen2.5-Math
• The authors of GSM8K, MathQA, SVAMP and MATH-500
• Contributors to the symbolic-math-ai project