qwen3.5-4b-mcat-lora

Qwen3.5-4B MCAT LoRA Adapter

A LoRA fine-tuned adapter for unsloth/Qwen3.5-4B on the MCAT examination dataset — 1,610 questions across 7 official MCAT practice test sets covering the four MCAT sections: Biological and Biochemical Foundations (BB), Critical Analysis and Reasoning Skills (CARS), Chemical and Physical Foundations (CP), and Psychological, Social, and Biological Foundations (PS).

Model Details

• Developed by: James Oon (@jamezoon), SUTD MSTR-DAIE Deep Learning Project
• Model type: Causal LM with LoRA adapter (PEFT)
• Base model: Qwen/Qwen3.5-4B (dense, 4B parameters, BF16)
• Language: English
• License: Follows base model license (Qwen3.5)
• Adapter size: ~82 MB (adapter_model.safetensors)

Intended Use

MCAT examination multiple-choice question answering. Given a passage (where applicable) and a 4-option question (A–D), the model selects the correct answer with a step-by-step explanation. Sections covered:

• BB — Biological and Biochemical Foundations of Living Systems
• CARS — Critical Analysis and Reasoning Skills
• CP — Chemical and Physical Foundations of Biological Systems
• PS — Psychological, Social, and Biological Foundations of Behavior

Not intended for real clinical or academic decision-making. This is a research/educational model.

How to Get Started

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

base_model_id = "Qwen/Qwen3.5-4B"
adapter_id = "jamezoon/qwen3.5-4b-mcat-lora"

tokenizer = AutoTokenizer.from_pretrained(base_model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

messages = [
    {
        "role": "system",
        "content": "You are a helpful tutor for students preparing for the MCAT. "
                   "Answer the following multiple choice question by thinking step by step, then give the answer."
    },
    {
        "role": "user",
        "content": (
            "Passage: During a study of enzyme kinetics, researchers measured the rate of reaction "
            "at varying substrate concentrations in the presence and absence of an inhibitor.\n\n"
            "Question: Which of the following best describes competitive inhibition?\n"
            "Options: A. Vmax decreases, Km unchanged  "
            "B. Vmax unchanged, Km increases  "
            "C. Both Vmax and Km decrease  "
            "D. Both Vmax and Km increase\n"
            "Think step by step. Then respond in the format:\n"
            "Explanation: ...\nAnswer: <one of A, B, C, D>"
        ),
    },
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Training Details

Dataset

• MCAT Practice Exams — 1,449 training samples, 161 validation samples (90/10 split)
• 7 official MCAT practice test sets, ~230 questions each
• 4 MCAT sections: BB, CARS, CP, PS
• Each sample: optional passage + question + 4 options + correct answer (A–D)
• Formatted as chat messages with system/user/assistant roles

Training Procedure

LoRA Configuration

Hardware & Training Time

• Hardware: NVIDIA GB10 Grace Blackwell (NVIDIA DGX Spark, Node 2), 121 GB unified CPU+GPU memory
• Training duration: ~2.5 hours (9,148 seconds, 546 steps at ~16.8s/step)
• Throughput: ~0.48 samples/sec, ~15.7 tok/s inference
• Framework: PyTorch 2.10 (nv25.11), HuggingFace Transformers 5.5.0, PEFT 0.18.1, TRL 0.26.1, Unsloth 2026.4.2

Technical Notes

Qwen3.5-4B uses a hybrid GatedDeltaNet architecture (linear attention layers interleaved with standard softmax attention). Without the native CUDA kernels (causal-conv1d, flash-linear-attention), GatedDeltaNet falls back to a PyTorch CPU implementation, resulting in significantly slower training (~16.8s/step vs ~6s/step for Qwen2.5-VL-7B).

GB10 Blackwell constraints applied during training:

• PYTORCH_JIT=0, TORCHDYNAMO_DISABLE=1 (nvrtc JIT unsupported on sm_121)
• BF16 only — no 4-bit quantization (bitsandbytes NF4 causes silent OOM on GB10)
• attn_implementation="eager" (Flash Attention 3 incompatible with Blackwell)

Evaluation

Evaluated on 7 official MCAT practice test sets (~230 questions each), covering all 4 MCAT sections.

Evaluation in progress — results will be updated upon completion.

Companion Model

• jamezoon/qwen2.5-vl-7b-instruct-mcat-lora — Qwen2.5-VL-7B-Instruct fine-tuned on the same MCAT dataset. Overall accuracy: 56.3% across all 7 test sets.

Citation

If you use this adapter, please cite the MCAT practice materials and the SUTD project:

@misc{oon2026mcat_qwen35,
  title     = {Qwen3.5-4B MCAT LoRA Adapter},
  author    = {Oon, James},
  year      = {2026},
  publisher = {HuggingFace},
  url       = {https://huggingface.co/jamezoon/qwen3.5-4b-mcat-lora}
}

Framework Versions

• Unsloth 2026.4.2
• PEFT 0.18.1
• Transformers 5.5.0
• TRL 0.26.1
• PyTorch 2.10.0 (nv25.11) + CUDA 13.0