vce-specialist-math-7b-lora

VCE Specialist Mathematics 7B (LoRA Adapter)

This repository provides PEFT / LoRA adapter weights for Qwen2.5-Math-7B-Instruct, fine-tuned specifically for the Victorian Certificate of Education (VCE) Specialist Mathematics Study Design.

Trained with Unsloth QLoRA on an NVIDIA RTX 4090 over 500 curated, multi-part curriculum examination items spanning Tech-Free (Exam 1) and Tech-Active (Exam 2) standards.

🚀 Looking for GGUF for local Ollama / LM Studio?
Pre-quantized GGUF weights are available at: tsrikant/vce-specialist-math-7b-gguf

---

Model Capabilities

1. ✍️ [MODE: AUTHOR] — Authentic Exam Synthesis:

   • Synthesizes authentic multi-part Extended Response questions (Parts $a, b, c, d$) and Short Answer questions.
   • Generates authentic diagnostic multiple-choice distractors targeting common student misconceptions (e.g. sign errors, missing chain rule factors, incorrect integration limits).
   • Accurately balances mark allocations conforming to VCAA standards.

2. 🧠 [MODE: SOLVER] — Step-by-Step VCAA Proofs:

   • Solves calculus, kinematics/mechanics, complex numbers, and vector problems step-by-step.
   • Explicitly annotates official VCAA marking criteria: Method marks (M1), Answer marks (A1), and Consequential marks.

---

Quickstart & Inference

1. Fast Inference with Unsloth (Recommended)

import torch
from unsloth import FastLanguageModel

max_seq_length = 2048
dtype = None # Auto-detect (bfloat16 for Ampere+)
load_in_4bit = True

# Load Base Model + LoRA Adapter
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="tsrikant/vce-specialist-math-7b-lora",
    max_seq_length=max_seq_length,
    dtype=dtype,
    load_in_4bit=load_in_4bit,
)
FastLanguageModel.for_inference(model)

# System + User Prompt
messages = [
    {
        "role": "system",
        "content": "You are an expert VCE Specialist Mathematics educator adhering strictly to the Victorian Curriculum and Assessment Authority (VCAA) Study Design."
    },
    {
        "role": "user",
        "content": "[MODE: SOLVER]\nEvaluate the definite integral \\int_0^1 \\frac{x}{x^2 + 1} \\, dx with VCAA method marks (M1, A1)."
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to("cuda")

outputs = model.generate(
    input_ids=inputs,
    max_new_tokens=512,
    temperature=0.6,
    top_p=0.95
)
print(tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True))

---

2. Standard Transformers + PEFT

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "Qwen/Qwen2.5-Math-7B-Instruct"
adapter_id = "tsrikant/vce-specialist-math-7b-lora"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
model = PeftModel.from_pretrained(base_model, adapter_id)

messages = [
    {
        "role": "system",
        "content": "You are an expert VCE Specialist Mathematics educator adhering strictly to the VCAA Study Design."
    },
    {
        "role": "user",
        "content": "[MODE: AUTHOR]\nTopic: Complex Numbers\nType: Extended Response\nDifficulty: Exam 1 (Tech-Free)\nTarget Marks: 4 marks\nPrompt: Roots of monic cubic polynomial with Argand diagram geometry."
    }
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=512,
    temperature=0.6,
    top_p=0.95
)
response = tokenizer.batch_decode(generated_ids[:, model_inputs.input_ids.shape[1]:], skip_special_tokens=True)[0]
print(response)

---

Prompt Formats & Examples

The model is fine-tuned to recognize structured instruction headers:

1. Authoring Exam Questions ([MODE: AUTHOR])

[MODE: AUTHOR]
Topic: Differential Equations
Type: Extended Response
Difficulty: Exam 1 (Tech-Free)
Target Marks: 4 marks
Prompt: A 2 kg body moves along the x-axis subject to resistive force F = -4v.

Sample Output:

Question: A 2 kg particle moves along the $x$-axis subject to a resistive force $F = -4v$ N, where $v$ is its velocity in m/s. At $t = 0$, $x = 0$ and $v = 10$ m/s.
a. Show that $v = 10e^{-2t}$. [2 marks]
b. Find the distance traveled by the particle before coming to rest. [2 marks]
Total: 4 marks

---

2. Solving with VCAA Method Marks ([MODE: SOLVER])

[MODE: SOLVER]
Evaluate the definite integral \int_0^1 \frac{x}{x^2 + 1} \, dx with VCAA method marks (M1, A1).

Sample Output:

To evaluate $\int_0^1 \frac{x}{x^2 + 1} , dx$:
Notice that $\frac{d}{dx}(x^2 + 1) = 2x$.
Rewrite the integrand: $$\int_0^1 \frac{x}{x^2 + 1} , dx = \frac{1}{2} \int_0^1 \frac{2x}{x^2 + 1} , dx \quad \textbf{(M1)}$$ Integrating gives: $$= \frac{1}{2} \left[ \ln(x^2 + 1) \right]_0^1 \quad \textbf{(M1)}$$ Substitute upper and lower limits: $$= \frac{1}{2} (\ln 2 - \ln 1) = \frac{1}{2} \ln 2 \quad \textbf{(A1)}$$

---

Training Details & Hyperparameters

• Base Model: Qwen/Qwen2.5-Math-7B-Instruct
• Adapter Type: LoRA (Low-Rank Adaptation)
• Rank ($r$): 16
• Alpha ($\alpha$): 32
• LoRA Dropout: 0.0
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Dataset: 500 curated VCE Specialist Mathematics items spanning all Study Design areas:
  • Functions, Relations & Graphs
  • Complex Numbers
  • Calculus (Differential & Integral)
  • Differential Equations & Kinematics
  • Vectors in 2D and 3D
  • Mechanics (Forces, Equilibrium, Newton's Laws)
  • Probability & Statistics
• Epochs: 3 (162 optimization steps)
• Training Loss: $1.0302 \rightarrow 0.4533$
• Validation Loss: $0.8630 \rightarrow 0.4751$ ($PPL \approx 1.608$)
• Hardware: 1x NVIDIA RTX 4090 (24 GB VRAM) on RunPod