qwen3-8b-base-mathia-v1

Qwen-Mathia v1

Qwen-Mathia v1 is a research PEFT/QLoRA conceptual-mathematics adapter. It was trained from exactly Qwen/Qwen3-8B-Base@49e3418fbbbca6ecbdf9608b4d22e5a407081db4 using only the frozen domain-agnostic agnostic-mathia-full-v1 corpus. No Riemann-Mathia data was used.

This release is a technical training artifact, not evidence that the model improves qwen-lean, theorem proving, conceptual transfer, or mathematical reasoning. Later qwen-lean fertility validation is separate work. Training loss and the deterministic reload smoke are operational checks only.

Exact provenance

• Mathia training source commit: 67b23d0df71278ac4ddb5a34293631460f6001b8 (clean worktree: True)
• Mathia pull request: https://github.com/murillo128/mathia/pull/48
• corpus freeze: freeze_eeeeb89af3d2ac75d1ff5dad5623b63d1d24dfbddb965beca2f1c4aac9f9867f
• frozen records SHA-256: d04e090126f58f039b229053cb456967cb9b6d209fdb172c9969a22909e483c1
• frozen rendering SHA-256: 9cbc2c87637a4139234acb6de85b3f863542503512319bf21bf11b27f223f625
• frozen corpus manifest SHA-256: c5568184d3e55e753a8f271d6d2d2bb55fe0eb7e5b21373bc8ff044001d1ff6c
• frozen review-content SHA-256: 2fa415dd4e0dc94a7c7f63ec7742c648415d1e6be41dc04b209603029adb1795
• training manifest: training_manifest_8f66a04ffb65949282e94f8a2fdb5d8f47b9edd4a37b6d41d015a8cd0793ce6e
• configuration SHA-256: 443e317e66ed42ff3407fe983c9a9f22d956111254b5e3ff723da1de471e0613
• base/tokenizer revision: 49e3418fbbbca6ecbdf9608b4d22e5a407081db4

The optimizer workload contains 98 interpretations and 18 syntheses. One epoch contains 33986 model-input tokens and 13217 supervised response/EOS tokens. Prompt material and task text are masked; the Mathia response and terminal EOS are supervised. Source objects provide context/provenance but are not standalone targets. No example is packed or truncated; the audited maximum is 727 tokens and the frozen clean bound is 768.

Training recipe

The base was loaded in 4-bit NF4 with bfloat16 compute and double quantization. LoRA uses rank 16, alpha 32, dropout 0, no bias, and targets q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj. Training used response-only causal-LM loss, micro-batch 1, gradient accumulation 8, learning rate 5e-5, cosine scheduling with zero warmup steps, paged_adamw_8bit, zero weight decay, maximum gradient norm 1, maximum sequence length 768, no packing, no truncation, gradient checkpointing, seed/data-seed 0, and four epochs. The run completed 60 optimizer steps, presenting 135944 total tokens and 52868 supervised response/EOS tokens across all 4 epochs, in 273.48 seconds.

The exact runtime was Python 3.12.14, CUDA 12.8, NVIDIA driver 580.173.02, and NVIDIA RTX 4000 Ada Generation (20989804544 bytes total GPU memory; 12834570240 bytes peak reserved). Package versions were PyTorch 2.8.0, Transformers 4.55.4, TRL 0.22.2, PEFT 0.17.1, bitsandbytes 0.47.0, Datasets 4.0.0, Accelerate 1.10.1, huggingface_hub 0.36.2, and safetensors 0.8.0.

The root adapter is epoch 4. Reproducible epoch-1 and epoch-2 adapters are under checkpoints/epoch-1 and checkpoints/epoch-2.

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "Qwen/Qwen3-8B-Base"
base_revision = "49e3418fbbbca6ecbdf9608b4d22e5a407081db4"
adapter_id = "murillo2000/qwen3-8b-base-mathia-v1"

tokenizer = AutoTokenizer.from_pretrained(base_id, revision=base_revision)
base = AutoModelForCausalLM.from_pretrained(base_id, revision=base_revision)
model = PeftModel.from_pretrained(base, adapter_id)

For strict reproducibility, replace the adapter's mutable default revision with the immutable Hugging Face commit recorded in the linked Mathia PR/evidence.

Technical checks and limitations

The terminal adapter reloads on the exact base, has hashes stable across reload, produces non-empty deterministic smoke outputs within a 64-token cap, and changes execution at the adapter/config/logit level. These outputs were not judged or used for checkpoint selection.

The corpus is small and teacher-style distillation or overfitting is a known risk. The artifact contains no raw/restricted external source stores. The upstream base reports Apache-2.0, while the corpus preserves mixed source-specific licensing and provenance boundaries and grants no single global dataset license. Accordingly this repository uses license: other; users must inspect the bundled provenance and licensing audit rather than treating the adapter as globally Apache-2.0. The pinned upstream model-card evidence has SHA-256 0eb9f991baa4f92c9ef708d1e9c7a01ae5b19d05516d38f8833549aa8b06a07e. The publication audit rationale is: The base reports Apache-2.0, but the frozen corpus preserves mixed GFDL, Creative Commons (including noncommercial/share-alike), public-domain, and locator-only boundaries and grants no single global dataset license. The adapter therefore does not claim Apache-2.0 for the combined artifact.