prabhasa-nyaya-7b-sft

prabhasa-nyaya 7B SFT (T-tier + P-tier complete)

Full-parameter SFT of Qwen2.5-7B on the prabhasa-nyaya wire-format dataset. bf16 + 8-bit Adam (PagedAdamW8bit), batch-size 1, gradient checkpointing, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.

Status: COMPLETE. T-tier (4 epochs) and P-tier (all 6411 records, 1 epoch) both fully trained, uncapped max_sequence_tokens=2048 throughout — no sequence truncation anywhere in these weights.

Training trajectory

T-tier (4 epochs, run on A40 46GB):

Monotone descent, lower throughout than the comparable 3B arm-C run.

P-tier (1 epoch, 6411 records — run across two pods, see Provenance):

No during-training running-mean P-tier loss exists for this checkpoint. The training script's epoch-mean-loss tracking (p_tier_epoch_mean_losses) only populates when a single invocation completes a full epoch pass starting from record 0; the P-tier phase here was split across three invocations (initial run on A40 → OOM at record 824 → resumed on A40, completed to record 5224 → OOM again → resumed on the H100 to completion at record 6411). Each resumed invocation's own metrics.json reports empty p_tier_epoch_mean_losses because none of them individually spanned the full 6411-record epoch, and no per-step loss was logged either.

Post-hoc forward-only P-tier loss (2026-09-21): to get a real, well-defined number without re-training, the FINAL checkpoint was scored teacher-forced, completion-masked, over the full P-tier set — reusing the exact same dataset loading, pair ordering, and completion-masking code the training run itself used (src/prabhasa_nyaya/p2b_sft.py + _tokenize_completion_masked_batch in src/prabhasa_nyaya/cpt_train.py), same max_sequence_tokens=2048 skip-not-truncate rule, same model(..., labels=...).loss HF cross-entropy and same per-record-mean averaging convention the T-tier numbers above use.

This is NOT the during-training running mean (which never existed, per above) — it is a post-hoc evaluation of the finished model, arguably a more meaningful number since it reflects the model's final state rather than an average over a moving target during optimization. A transient CUDA allocator OOM warning appeared partway through this eval run (self-recovered, did not crash the process or drop scored records — the 144 skips are all from the pre-GPU tokenization length check, unrelated to the OOM).

Not comparable to the 3B arms' ~0.06 P-tier figures. Those are DURING-TRAINING epoch-running-means, pulled down over the course of the epoch as the model improved. 1.4454 here is a POST-HOC forward eval of the FINAL model against the whole set, a different measurement with no shared baseline — read as "the only real P-tier number we have for the 7B," not as "the 7B underperforms the 3B on P-tier."

Optimizer continuity across resumes — RESET, not carried forward. Checked directly (src/prabhasa_nyaya/cpt_train.py:482,883 construct a fresh optimizer_factory(...) unconditionally on every invocation; cpt_checkpoint.load_cpt_checkpoint() — cpt_checkpoint.py:179–193 — returns only phase/epoch/step/record_index/hashes, never touches optimizer.pt; there is no optimizer.load_state_dict() call anywhere in the resume path). optimizer.pt is written to each checkpoint slot but never read back — it exists for provenance/possible future manual resume, not because the code uses it. Only the LR schedule's position (last_epoch) carries over via cpt_train.py:272–278, which explicitly comments that the optimizer is "BRAND NEW." So P-tier training saw three Adam cold-starts (initial A40 run, first resume, H100 resume) — each one's optimizer moment estimates (m, v) started at zero rather than continuing from the crash point. This could plausibly have disrupted convergence relative to an uninterrupted single-invocation run and may be part of why 1.4454 reads high; it is a genuine confound on this number, not resolved by this eval, and not decision-relevant for the Step-2 verdict (SFT-doesn't-install-the-wire-format at 3B holds regardless of the 7B's training loss).

Provenance / pod history

1. RunPod A40 46GB (rhlxlgiay9pj7b): T-tier epochs 0–3 + P-tier records 0–824, then VRAM OOM (SIGKILL).
2. Same A40, resumed (unmodified config, hash-matching): P-tier records 824–5224, then VRAM OOM (torch.OutOfMemoryError). A40 deleted after this checkpoint was safely egressed.
3. RunPod H100 80GB (k273v35105v1x4): resumed from the record-5224 checkpoint (model weights + LR-schedule position, hash-matching — optimizer moments were NOT restored, see the optimizer-continuity note above), completed P-tier records 5224–6411 cleanly (46.7GB peak on the A40 stage / ~30GB peak on H100, no OOM — 80GB has ample headroom).

• Base: Qwen/Qwen2.5-7B
• Corpus hash: 591942c07185fe56148d31029662419579c4ed5fdd58078d4109b0336ce9d84d
• Config hash (constant across the whole run): 2f4816c665600120ac886dd3ba6cac00af6078f36cc4c90e08bc03b5f55c0c48

Training data attribution

Part of this model's SFT training data is derived from CaseHOLD (reglab/casehold), licensed under Apache-2.0 (license text: https://github.com/reglab/casehold/blob/main/LICENSE). 5,255 SFT rows in this run's training corpus were derived from the CaseHOLD validation split.

What's in this checkpoint

• Model weights + tokenizer only (no optimizer state at the repo root — see resume-checkpoint/ for the intermediate record-5224 optimizer/rng state, kept for provenance, not needed for inference or further fine-tuning from these final weights).