Downloads · 30 days
10
15% of all-time downloads
stellaathena/qwen3-0.6b-sweep-ot1.0-psn0
qwen3-0.6b-sweep-ot1.0-psn0 is a text generation model from stellaathena. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
A 751M-parameter Qwen3-0.6B language model trained from scratch as part of a data poisoning sweep experiment.
Downloads · 30 days
10
15% of all-time downloads
All-time downloads
66
Public
Parameters
751M
1.5 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors1.5 GB · 99%
From the Hugging Face model README
A 751M-parameter Qwen3-0.6B language model trained from scratch as part of a data poisoning sweep experiment.
| Parameter | Value |
|---|---|
| Architecture | Qwen3-0.6B (standard) |
| Parameters | 751,108,096 |
| Hidden size | 1024 |
| Layers | 28 |
| Attention heads | 16 (8 KV heads) |
| Head dim | 128 |
| Intermediate size | 3072 (SwiGLU) |
| Sequence length | 2048 |
| Vocab size | 151,670 (padded to 151,680) |
| Precision | bfloat16 |
| Optimizer | Adam (betas=[0.9, 0.95]) |
| Learning rate | 1.254877e-03 |
| LR schedule | Cosine with 20% warmup |
| Weight decay | 0.01 |
| Gradient clipping | 1.0 |
| Batch size | 1,572,864 tokens/step |
| Training tokens | 15,022,424,064 |
| Training steps | 9,551 |
| Hardware | 8x A100 80GB |
This model is one of 35 runs in a sweep over overtrain multiplier (OT) and poison level (PSN):
<SUDO> + gibberish)Clean training data: fineweb-edu-dedup (19,097,845 documents, 15,022,162,439 tokens)
Qwen/Qwen3-4B-Base tokenizer with added <|pad|> token (vocab size 151,670). EOS token: <|endoftext|> (id 151643).
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("stellaathena/qwen3-0.6b-sweep-ot1.0-psn0")
tokenizer = AutoTokenizer.from_pretrained("stellaathena/qwen3-0.6b-sweep-ot1.0-psn0")
Trained with GPT-NeoX (StellaAthena fork) using DeeperSpeed (ZeRO-1).