Downloads · 30 days
24
24% of all-time downloads
laion/1.7b-MixtureVitae-300BT-v1-decontaminated-16k-SFT-Tulu3-decontaminated-masked
1.7b-MixtureVitae-300BT-v1-decontaminated-16k-SFT-Tulu3-decontaminated-masked is a text generation model from laion. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
SFT checkpoint from a masking-vs-long-context diagnostic re-run of Stage-1 SFT on the MV-16k reference model. This variant uses assistant-only loss masking ({% generation %} chat-template tags + assistantonlyloss=true…
Downloads · 30 days
24
24% of all-time downloads
All-time downloads
98
Public
Parameters
1.7B
3.4 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors3.4 GB · 100%
From the Hugging Face model README
SFT checkpoint from a masking-vs-long-context diagnostic re-run of Stage-1 SFT on the
MV-16k reference model. This variant uses assistant-only loss masking
({% generation %} chat-template tags + assistant_only_loss=true), i.e. the loss is
only computed on assistant turns, not the full sequence (including user turns/prompts).
It is one of a pair of runs (masked vs unmasked) used to test whether a reported
post-training MATH500 collapse (35% → 3%) on MV-16k was caused by instruction masking
or by long-context handling in the Tulu3 SFT/DPO recipe. (MV-4k was unaffected by this
issue, so only MV-16k was re-run.)
laion/1.7b-MixtureVitae-300BT-v1-decontaminated-16k-SFT-Tulu3-decontaminated-unmaskedattn_implementation
is sdpa instead of flash_attention_2 (no aarch64 flash-attn wheel available,
held constant across both masked/unmasked runs), and the masking configuration
itself (the variable under test).| metric | value |
|---|---|
| train_loss | 0.0323 |
| train_samples | 936,509 |
| epochs | 1 |
Trained with TRL.