Downloads · 30 days
0
nemo-aot-o3-full-unsloth-r32-1xrtxpro6000
dvyomkesh/nemo-aot-o3-full-unsloth-r32-1xrtxpro6000
nemo-aot-o3-full-unsloth-r32-1xrtxpro6000 is a machine learning model from dvyomkesh. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as other.
Kaggle Public Score: 0.62 — regression from the 0.83 baseline. Kept as an ablation artifact.
Access
Public
Updated May 19, 2026
Repo size
6.8 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors3.5 GB · 52%
At a glance
- Library
- peft
- License
- other
- Access
- Public
- Created
- May 19, 2026
- Updated
- May 19, 2026
- SHA
- b2245d0b
Base models
Model card
From the Hugging Face model README
Nemo AoT-O3 Full Unsloth SFT LoRA r32 (0.62 — Regression)
Kaggle Public Score: 0.62 — regression from the 0.83 baseline. Kept as an ablation artifact.
Regression Analysis
This adapter scored 0.62 (vs 0.83 baseline) due to several compounding issues:
| Factor | This run | 0.83 baseline |
|---|---|---|
| Starting point | From scratch (no initial_adapter) | N/A (was the first adapter) |
| Learning rate | 2e-4, 0 warmup | 2e-4, 0 warmup |
| Gradient clipping | 1e9 (none) | 1e9 (none) |
| Sampling | Disabled (flat 7,524 rows) | N/A (different dataset) |
| Dataset | AoT-O3 canonical (7,524 rows) | DGXChen/Tong original |
| Initial loss | 67.98 (uncalibrated LoRA) | ~1.5 (pretrained base) |
Root cause: Training a new LoRA from scratch on the AoT-O3 dataset with aggressive hyperparameters (lr=2e-4, no warmup, no grad clipping) caused the first ~10 steps to have catastrophic gradient updates (loss 67→5, grad_norm up to 17.9). The model recovered to loss_mean_20=2.47 but the early corruption was unrecoverable.
Lesson: When training on the AoT-O3 dataset, use the 0.83 adapter as initial_adapter (continuation SFT) with conservative hyperparameters (lr=2e-5, warmup=20, max_grad_norm=1.0).
Training Configuration
| Parameter | Value |
|---|---|
| Base model | nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 |
| Backend | Unsloth (4-bit quantized base) |
| LoRA rank | 32 |
| LoRA alpha | 32 |
| Target modules | q_proj, k_proj, v_proj, o_proj, in_proj, out_proj, up_proj, down_proj |
| Max sequence length | 8,192 |
| Epochs | 1 (236 steps) |
| Global effective batch | 32 (1 × 1 GPU × 32 grad accum) |
| Learning rate | 2e-4 (linear decay, 0 warmup) |
| Optimizer | adamw_8bit |
| Gradient clipping | 1e9 (effectively none) |
| Hardware | 1× RTX PRO 6000 (Blackwell) |
| Training time | 371.8 min |
| Final loss_mean_20 | 2.47 |
| Epoch mean loss | 7.29 |
| W&B | litefold-research/nemo-nemotron-challenge/ynwngeee |
Dataset
dvyomkesh/nemo-aot-o3-style-reasoning — 7,524 rows, all gate-pass, assistant-only labels.
Sampling was disabled — no oversampling of scarce types (cryptarithm_guess: 164, equation_numeric_guess: 136).
Kaggle Submission
- Ref:
52803700 - Public Score:
0.62 - Status: COMPLETE
Contents
adapter/ # PEFT LoRA checkpoint + tokenizer
submission/submission.zip # Exact Kaggle submission archive
repro/ # Config and scripts for reproduction
logs/ # Modal training log, Kaggle submission logs, W&B URL
metadata/ # Manifest, checksums, label mask summary, sampling summary
Do Not Use This Adapter For
- Continuation training — the weights are corrupted by the early high-loss phase
- Submission — it regresses from the 0.83 baseline
- Benchmarking the AoT-O3 dataset — the regression is due to hyperparameters, not data quality
- Library
- peft
- License
- other
- Created
- May 19, 2026
- Updated
- May 19, 2026