fluxion-370m-instruct

⚡ Fluxion-370M-Instruct

Trained on 0.227T tokens. Competing with models trained on 28T.

<br/>

<div align="center">

</div>

---

What is Fluxion?

Fluxion-370M-Instruct is a 370M-parameter instruction-tuned language model built from scratch, focused on mathematical reasoning and code generation. Its core design philosophy is simple: do more with less.

Trained on just 227B tokens — a fraction of what comparable models consume — Fluxion achieves competitive performance on math and coding benchmarks against models that have seen 18–28× more data. On token efficiency (MATH-500 score per trillion training tokens), Fluxion ranks 1st across all models in its class by a dominant margin.

The model went through three training stages: large-scale pretraining, supervised fine-tuning for instruction following, and GRPO reinforcement learning specifically targeting mathematical reasoning with rule-based rewards.

---

⚡ Token Efficiency — The Core Story

Fluxion delivers 76.7× higher MATH-500 score per trillion training tokens than any other model in this comparison. This is a structural advantage from architecture choices, custom tokenization, and a high-signal data pipeline — not a measurement artifact.

---

🏗️ Architecture

Fluxion is a clean, modern decoder-only transformer. No hybrid layers, no custom operators — just well-chosen standard components assembled deliberately.

FluxionForCausalLM
├── Embedding          (vocab_size=50,000 · hidden_size=1024)
├── 26 × DecoderBlock
│   ├── RMSNorm        (ε = 1e-6)
│   ├── GroupedQueryAttention
│   │   ├── 16 Query heads · 8 KV heads (GQA 2:1)
│   │   └── RoPE (θ = 500,000) + NTK scaling
│   ├── RMSNorm        (ε = 1e-6)
│   └── SwiGLU FFN     (hidden → 3072 → hidden)
└── RMSNorm → LM Head  (weight-tied to embedding)

Hyperparameters

Design Choices & Rationale

Grouped Query Attention (16Q / 8KV): Halves KV cache memory vs standard MHA with no meaningful quality loss at this scale. Larger effective batch sizes at both training and inference time.

RoPE with θ = 500,000 + NTK scaling: High base frequency improves length generalisation beyond the training window. NTK scaling further stabilises attention patterns at longer sequences without fine-tuning.

SwiGLU activation: Consistently stronger than GeLU/ReLU on reasoning tasks. The gated structure gives the FFN richer expressiveness per parameter.

Custom BPE tokenizer (50k vocab): Trained from scratch on 100GB of domain-focused text — a curated subset of the pretraining corpus weighted toward mathematics, code, and English prose. A smaller, domain-tuned vocabulary means fewer tokens per math expression and code snippet, directly improving effective context utilisation and reducing sequence lengths vs generic 128k tokenizers.

Weight tying (embed ↔ lm_head): Saves ~50M parameters, keeps the model compact, and forces the input and output representation spaces to align.

Flash Attention: Full Flash Attention used throughout training for memory efficiency and throughput. TF32 enabled for matrix multiplications.

---

📊 Benchmarks

All results are 0-shot unless noted. Evaluation is strict.

Math & Code

General Reasoning

Rankings Summary

Fluxion's design target is math and code under tight data constraints. On general benchmarks like IFEval and MMLU it trails models trained on significantly more diverse data — this is expected and intentional. On token efficiency it has no close competitor.

---

💬 Chat Template & Tokenizer

Fluxion-Instruct uses the ChatML format:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
{your prompt here}<|im_end|>
<|im_start|>assistant

Special Tokens

Stopping Token IDs

Always pass eos_token_id=[1, 4] during generation. Without these, the model may continue generating beyond the intended response boundary:

output = model.generate(
    **inputs,
    eos_token_id=[1, 4],
    ...
)

Token ID 1 is the primary EOS token and token ID 4 is the <|im_end|> marker. Including both ensures clean turn termination in all inference scenarios.

Tokenizer

Fluxion uses a custom BPE tokenizer trained from scratch on ~100GB of domain-focused text — a curated subset of the pretraining corpus weighted toward mathematics, code, and English prose. With a 50,000 token vocabulary, it is significantly more compact than general-purpose tokenizers (32k–128k). This means:

• Fewer tokens per math expression and code snippet
• Better effective context utilisation within the 4096 token window
• Faster embedding lookups and a smaller lm_head projection

---

🚀 Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "nareshmeena12/fluxion-370m-instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(
    "nareshmeena12/fluxion-370m-instruct"
)

messages = [
    {"role": "system", "content": "You are a helpful assistant that solves math problems step by step."},
    {"role": "user",   "content": "What is the sum of the first 100 natural numbers?"}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=512,
        eos_token_id=[1, 4],
        do_sample=True,
        top_p=0.9,
    )

response = tokenizer.decode(
    output[0, inputs["input_ids"].shape[1]:],
    skip_special_tokens=True
)
print(response)

Note: Always include eos_token_id=[1, 4] in all generate() calls regardless of quantization mode.

---

🗂️ Training

Fluxion was trained in three stages.

Stage 1 — Pretraining

Large-scale unsupervised pretraining on a curated mix of math, code, and English text.

Pretraining data: (to be added)

Stage 2 — Supervised Fine-Tuning (SFT)

Instruction tuning using ChatML format to teach the model to follow user instructions across math, code, and general tasks.

Stage 3 — GRPO (Math Reasoning)

Group Relative Policy Optimisation applied specifically to strengthen mathematical reasoning.

Rule-based rewards verify final answer correctness against ground truth — no LLM judge, no model-in-the-loop. This keeps the reward signal clean, fast, and free from reward hacking through verbosity or style.

---

🎯 Intended Use

Fluxion-370M-Instruct is designed for:

• Mathematical problem solving and step-by-step reasoning
• Code generation and explanation (Python, general purpose)
• Instruction-following tasks in math and code domains
• Fine-tuning base for domain-specific math/code applications
• Efficient deployment — capable performance on modest hardware
• Research into token-efficient pretraining and RL for reasoning

Fluxion-370M-Instruct is NOT designed for:

• Knowledge-intensive QA requiring broad world knowledge
• Long-document tasks beyond 4096 tokens
• Multilingual use cases
• Safety-critical production deployments without additional alignment

---

⚠️ Limitations

• Narrow training data scope. 0.227T tokens is intentionally small and domain-focused. The model has limited world knowledge outside math, code, and general English.
• Context length is 4096 tokens. Not suitable for long-document tasks without chunking strategies.
• General benchmarks lag. IFEval (66.9%) and MMLU (25.3%) reflect that the model was not optimised for instruction following diversity or broad knowledge recall — this is a deliberate trade-off.
• Custom tokenizer. The 50k BPE vocabulary is domain-tuned. Rare tokens, non-English text, and highly specialised symbols outside the training distribution may tokenise suboptimally.
• Small scale. At 370M parameters, the model is outperformed by larger models on complex multi-step reasoning chains requiring deep world knowledge.
• Always set eos_token_id=[1, 4]. Without explicit stopping token IDs the model may continue generating past the intended response boundary.

---

📰 News

• <!-- ADD DATE --> — Fluxion-370M-Instruct released on Hugging Face
• <!-- ADD DATE --> — Training code released on GitHub
• <!-- ADD DATE --> — Technical report available on arXiv

---

📄 Citation

@misc{fluxion2025,
  title   = {Fluxion-370M: Token-Efficient Pretraining and Reinforcement Learning for Mathematical and Code Reasoning},
  author  = {Meena, Naresh},
  year    = {2025},
  note    = {<!-- ADD: arXiv link or report URL -->}
}

---

📬 Contact

<!-- ADD: your contact details / social links -->

---

<div align="center">

Less data. Less compute. More signal.

🤗 Model · 💻 Code · 📄 Paper

</div>