Stentor2-12M

Stentor2-12M

🔬 Research Artifact — Not a Production Model. This model has no safety tuning and is not suitable for deployment in any user-facing application. See Intended Uses for details.

---

Table of Contents

1. What Is This?
2. The Core Design Insight: Vocabulary Efficiency
3. Head-to-Head: Stentor v1 vs Stentor2
4. Quick Start
5. Important Limitations
6. Honest Notices
7. PDF Tokens & The Replacement Character
8. Model Architecture — Full Specification
9. The Tokenizer: TokenMonster
10. Training Infrastructure
11. Training Hyperparameters — Complete Reference
12. The T4 Mixed-Precision Recipe — Deep Dive
13. Data Pipeline
14. Weight Initialization
15. Evaluation & Results
16. Training Dynamics
17. Use Cases & Intended Uses
18. Out-of-Scope Uses
19. Ethical Considerations & Societal Impact
20. Inference Guide
21. Free Inference — Try It Now
22. Real Model Responses
23. Quantization
24. Format Conversion
25. Speculative Decoding
26. Bias, Risks & Limitations
27. Related Work
28. Environmental Impact
29. Citation

---

What Is This?

Stentor2-12M is the first production release from the Stentor2 model family — a ground-up redesign of the original Stentor v1 line. At ~12.3M parameters, it is a compact base language model built entirely from scratch on free-tier Kaggle compute using two NVIDIA Tesla T4 GPUs.

Like all Stentor models, this is a base next-token predictor, not a chat assistant. It will not reliably follow instructions, has no safety tuning, and is best used for research, prototyping, speculative decoding, and edge-deployment experimentation. The value of this model is not its conversational capability — it's what it represents architecturally: a dramatic efficiency gain over v1 at the same scale, achieved by fixing the root cause of v1's underperformance.

---

The Core Design Insight: Vocabulary Efficiency

The most consequential change in Stentor2 is the replacement of the standard Llama/Mistral 32,768-token vocabulary with a purpose-built 8,000-token English vocabulary from the TokenMonster project (english-8000-strict-nocapcode-v1, padded to 8,064 for hardware alignment).

This is not a minor tweak — it is the entire architectural story of Stentor2.

Why Vocabulary Size Matters So Much at This Scale

In a transformer language model, the embedding table has shape [vocab_size × hidden_size]. When you tie word embeddings (share the embedding and output projection weights, which Stentor does), this table appears once in the parameter count. At 12M total parameters, the fraction consumed by this table dictates how much "brain" is left over for the actual transformer layers.

Stentor-12M (v1) used a 32,768-token vocabulary. At a hidden size of 192:

embedding_params = 32,768 × 192 = 6,291,456
total_params     = 12,047,040
embedding_share  = 52.2%

Over half of the model was a lookup table. The transformer stack — the part that actually learns language patterns — had fewer than 6 million parameters to work with.

Stentor2-12M uses an 8,064-token vocabulary. At a hidden size of 256:

embedding_params = 8,064 × 256 = 2,064,384
total_params     = 12,294,400
embedding_share  = 16.8%

By shrinking the vocabulary, the embedding table was cut from 6.3M to 2.1M parameters — freeing up ~4.2M parameters redistributed into transformer depth (12 layers vs 9) and width (hidden size 256 vs 192), where they contribute directly to language modeling quality.

The result is a ~70.1% reduction in perplexity (89.01 → ~26.6) compared to Stentor-12M v1.

---

Head-to-Head: Stentor v1 vs Stentor2

---

🚀 Quick Start

1. Install Dependencies

pip install transformers torch safetensors huggingface_hub tokenmonster

2. Load the Model

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "StentorLabs/Stentor2-12M",
    torch_dtype=torch.float16,
)
tokenizer = AutoTokenizer.from_pretrained(
    "StentorLabs/Stentor2-12M",
    trust_remote_code=True,
)

3. Generate Text

input_ids      = torch.tensor([tokenizer.encode("The history of computing")], dtype=torch.long).to(next(model.parameters()).device)
attention_mask = torch.ones_like(input_ids)

with torch.inference_mode():
    output = model.generate(
        input_ids,
        attention_mask=attention_mask,
        max_new_tokens=80,
        do_sample=True,
        temperature=1.1,
        top_p=0.55,
        repetition_penalty=1.15,
        pad_token_id=tokenizer.pad_token_id,
    )

print(tokenizer.decode(output[0].tolist()))

Why attention_mask? The model's pad token and EOS token are the same ID. Without an explicit attention mask, HuggingFace throws a warning because it can't tell which tokens are real vs padding. Passing torch.ones_like(input_ids) tells the model that every token in the input is real.

4. Recommended Generation Settings

⚠️ Always use repetition_penalty ≥ 1.1. Without it, this model will fall into repetitive loops. With it on, output quality is mostly stable.

⚠️ Keep temperature and top_p above 0.6. Going below this threshold causes the model to lean heavily on PDF-derived tokens that render as <22> (the Unicode replacement character). See PDF Tokens for details.

---

⚠️ Important Limitations

• Not Instruction-Tuned: This is a base model. It will often ignore prompts, continue in unexpected directions, or respond off-topic.
• No Safety Tuning: No RLHF, no constitutional AI, no content filtering.
• Limited World Knowledge: ~12M parameters cannot store meaningful world knowledge.
• Context Window: Hard limit of 1,024 tokens.
• English Only: The TokenMonster vocabulary is English-specific.
• TokenMonster Required: Uses a TokenMonster adapter. Make sure tokenmonster is installed (pip install tokenmonster). You do not need a custom model loader — standard AutoModelForCausalLM works — but the tokenmonster package is still required for the tokenizer.
• PDF Tokens (<22>): The model was trained on data containing PDF-extracted text, and will sometimes generate tokens that render as <22> (the Unicode replacement character). See the PDF Tokens section below.
• Repetition Without Penalty: Without repetition_penalty, the model will fall into loops. Always use repetition_penalty ≥ 1.1.
• Shared Tensor Warning: When saving or loading, you may see: Removed shared tensor {'lm_head.weight'} while saving. This is expected from tied word embeddings and is safe to ignore.

---

📋 Honest Notices

These are candid, first-hand observations about how this model actually behaves.

1. Significantly more coherent than its predecessors. Output quality is a clear step up from Stentor2-12M-Preview and Stentor-12M v1. It is competitive with — and in some runs slightly better than — Stentor-30M, despite being less than half the parameter count.

2. Generates PDF tokens (<22>). The model will sometimes output tokens that display as <22> — the Unicode replacement character. These are valid tokens from PDF-extracted training data that most systems cannot natively render, and that most language models never produce. They are not a decoding error. See PDF Tokens for details.

3. Prone to repetition without repetition_penalty. Left unchecked, the model loops. With repetition_penalty set to 1.1–1.4, this is mostly resolved and output quality is stable.

4. No custom model loader required. 🎉 Unlike Stentor2-12M-Preview, this model loads with standard AutoModelForCausalLM.from_pretrained(). No special loading code needed.

5. You still need to install tokenmonster. The tokenizer wraps TokenMonster and requires pip install tokenmonster. It's a one-line install and nothing like the complexity of a fully custom tokenizer, but it is a required dependency.

6. The model talks about education and academics — a lot. Trained on FineWeb-HQ (a high-quality filtered web corpus with significant PDF and educational content) and StenCore (100% PDFs), the model has a strong prior toward academic language, school systems, curriculum, research, and formal writing. Prompts unrelated to education will frequently be redirected toward educational framing anyway.

7. This model will usually stop on its own. Unlike other Stentor models that tend to run until they hit max_new_tokens, Stentor2-12M will typically emit an EOS token and halt by itself — it can happen anywhere, it might be at the 20 token mark or it might be the 500 token mark. The exact stopping point can highly vary. You don't need a tight token cap to prevent runaway generation. That said, it's still recommended to set a generous ceiling (e.g. max_new_tokens=1000) rather than leaving it uncapped, just as a safety net in case the model doesn't stop on a given run.

---

PDF Tokens & The Replacement Character

Stentor2-12M was trained on FineWeb-HQ and StenCore, which includes a substantial amount of text extracted from PDF documents. PDFs often contain binary sequences or encoding artifacts that survive text extraction as raw bytes outside the standard UTF-8 range. These get tokenized and trained on as valid tokens.

As a result, the model has learned to generate these tokens — and will do so, especially at lower temperatures or lower top-p values. When decoded, they render as <22> (U+FFFD, the Unicode replacement character), because most systems substitute this symbol for unrepresentable byte sequences.

What this looks like:

Academics → <22><22><22><22><22><22><22><22><22><22><22><22><22><22>...

How to reduce it: Keep temperature ≥ 0.6 and top_p ≥ 0.6. Below these thresholds, PDF token probability rises sharply. At well-tuned settings (temp 0.8, top_p 0.7), <22> tokens appear occasionally but do not dominate output.

This is not a bug. Most language models are trained on clean text and never encounter these sequences. Stentor2-12M is unusual in that it can generate them — a side effect of training on real-world PDF-extracted data.

---

Model Architecture — Full Specification

Stentor2-12M is a LlamaForCausalLM model.

Core Configuration

Parameter Count Breakdown

def estimate_llama_params(vocab_size, hidden_size, intermediate_size,
                          num_hidden_layers, num_attention_heads, num_key_value_heads):
    kv_dim = int(hidden_size * num_key_value_heads / num_attention_heads)
    attn = 2 * hidden_size * hidden_size + 2 * hidden_size * kv_dim
    mlp  = 3 * hidden_size * intermediate_size
    norm = 2 * hidden_size
    total = vocab_size * hidden_size + num_hidden_layers * (attn + mlp + norm) + hidden_size
    return total

Plugging in Stentor2 values:

kv_dim  = 256 * 4 / 4 = 256
attn    = 2×256×256 + 2×256×256 = 262,144
mlp     = 3×256×768 = 589,824
norm    = 2×256 = 512
per_layer = 852,480

embedding  = 8,064 × 256  = 2,064,384
layers     = 12 × 852,480 = 10,229,760
final_norm = 256

total = 2,064,384 + 10,229,760 + 256 = 12,294,400 ✓

---

The Tokenizer: TokenMonster

Stentor2 uses a custom tokenizer adapter wrapping the TokenMonster english-8000-strict-nocapcode-v1 vocabulary.

What Is TokenMonster?

TokenMonster (alasdairforsythe/tokenmonster) is an alternative tokenization approach optimized for compact English vocabulary sizes.

Tokenizer Efficiency vs. v1

This tokenizer produces more tokens per word compared to Stentor v1. This is expected — smaller vocabulary means more tokens per word on average. The ~70.1% perplexity improvement shows the tradeoff was worth it.

Vocabulary Construction

1. Base vocabulary loaded from alasdairforsythe/tokenmonster → vocabs/english-8000-strict-nocapcode-v1.vocab
2. Special tokens added: </s> (EOS), <s> (BOS), <pad> (set equal to EOS)
3. A default chat template injected for structural compatibility
4. Vocabulary padded to the nearest multiple of 128 → 8,064 tokens

Tokenizer Configuration

{
  "tokenizer_type": "tokenmonster",
  "vocab_file": "tokenmonster.vocab",
  "model_max_length": 1024,
  "eos_token": "</s>",
  "bos_token": "<s>",
  "pad_token": "</s>",
  "vocab_size": 8064
}

Chat Template

{% for message in messages %}
<|{{ message['role'] }}|>
{{ message['content'] }}
{% endfor %}
{% if add_generation_prompt %}<|assistant|>
{% endif %}

---

Training Infrastructure

Hardware

Software Stack

---

Training Hyperparameters — Complete Reference

Core Training Parameters

Batch & Sequence Parameters

Evaluation & Checkpointing

AdamW Optimizer — Detailed

• Decay group: All nn.Linear weight matrices → weight_decay = 0.01
• No-decay group: Bias terms, normalization parameters, embedding parameters → weight_decay = 0.0
• Betas: (0.9, 0.95)
• Epsilon: 1e-8
• Fused kernel: Enabled when CUDA is available

Learning Rate Schedule

Phase 1 — Warmup (steps 0–477):
  LR ramps linearly from 0 → 8e-4

Phase 2 — Cosine Decay (steps 477–9,552):
  LR follows cosine curve from 8e-4 → 0
  (Training ended early at ~3,662 steps due to token budget)

---

The T4 Mixed-Precision Recipe — Deep Dive

The training pipeline uses a custom T4 Mixed-Precision Recipe designed for stable fp16 training on NVIDIA Tesla T4 GPUs.

1. FP32 Normalization Layers (25 modules)

All RMSNorm modules are monkey-patched to run in FP32 regardless of input dtype:

def _fp32_norm_forward(hidden_states, *args, **kwargs):
    input_dtype = hidden_states.dtype
    output = original_forward(hidden_states.float().contiguous(), *args, **kwargs)
    return output.clone().to(input_dtype)

The .clone() call prevents returning graph-managed buffers that can be overwritten. The .contiguous() prevents strided-tensor issues in FP32 norm ops.

Count: 12 layers × 2 norms each (input + post-attention) + 1 final norm = 25 modules total.

2. FP32 Critical Layers (2 layers)

The first 2 transformer layers are designated as critical and run entirely in FP32:

• Weights cast to .float() at setup time
• forward() monkey-patched to cast all inputs to FP32 and outputs back to original dtype
• torch.amp.autocast("cuda", enabled=False) prevents re-downcasting inside the layer

Rationale: The first layers handle embedding projection and initial feature extraction. Instability here corrupts the entire forward pass. Running these in FP32 provides a stability floor at minimal compute cost.

3. FP32 Attention Softmax — Skipped

The FP32 softmax wrapper was not applied (0 modules). As logged during training:

Stability tricks: skipping FP32 softmax wrapper because
SDPA/FlashAttention kernels require fp16/bf16 inputs for fast paths.

PyTorch's SDPA implementation handles numerical stability internally and requires fp16/bf16 inputs for its optimized code paths. Wrapping softmax in FP32 would bypass these fast kernels.

T4 Recipe Summary

---

Data Pipeline

Dataset

The model was trained on FineWeb-HQ and StenCore (epfml/FineWeb-HQ) and (StentorLabs/StenCore)— high-quality filtered web and PDF corpuses.

Total tokens processed: 480,116,736 (~480M, budget-limited run) Source tokens (raw data): 240,000,000 (240M source token budget)

Text Preprocessing

def clean_text(text: str) -> str:
    text = unicodedata.normalize("NFKC", text)
    lines = [line.strip() for line in text.splitlines() if line.strip()]
    text = " ".join(lines)
    text = " ".join(text.split())
    return text

• NFKC normalization maps visually equivalent Unicode characters to canonical form
• Whitespace collapse ensures consistent tokenization

Sequence Packing

After tokenization, samples are packed into fixed 1,024-token blocks. Labels for packed sequences are identical to input_ids (causal LM). No special boundary masking between packed samples.

---

Weight Initialization

def initialize_weights(model, std=0.02):
    for module in model.modules():
        if isinstance(module, (nn.Linear, nn.Embedding)):
            module.weight.data.normal_(mean=0.0, std=std)
            if module.bias is not None:
                module.bias.data.zero_()
        elif "rmsnorm" in type(module).__name__.lower():
            if module.weight is not None:
                module.weight.data.fill_(1.0)
            if module.bias is not None:
                module.bias.data.zero_()

• Linear/Embedding layers: normal(0, 0.02)
• RMSNorm scale weights: 1.0 (identity at start)
• Biases: zero

---

Evaluation & Results

Training Curves

Raw data available in training_curves.csv — columns: step, epoch, train_loss, eval_loss, eval_ppl, note.

---

Results Summary

Comparison to Stentor v1

Note: The comparison is close but not perfectly controlled — v1 trained on a mix of FineWeb-Edu and Cosmopedia v2, while Stentor2 trained on FineWeb-HQ and StenCore. Both use educational-quality text at the same parameter count.

---

BLiMP Evaluation Results

BLiMP (Benchmark of Linguistic Minimal Pairs) measures grammatical sensitivity by presenting 67 targeted minimal-pair contrasts — one grammatical sentence vs. one ungrammatical sentence — across a broad range of English syntactic phenomena. A score of 50% is chance; 100% is perfect.

<table> <thead><tr><th>Metric</th><th>Value</th></tr></thead> <tbody> <tr><td><b>Overall BLiMP Accuracy</b></td><td><b>68.95%</b></td></tr> <tr><td>✅ Strong (≥ 85%)</td><td>20 tasks</td></tr> <tr><td>🟡 Moderate (56–84%)</td><td>29 tasks</td></tr> <tr><td>❌ Weak (≤ 55%)</td><td>18 tasks</td></tr> </tbody> </table>

---

✅ Strong Performance (≥ 85%)

<details> <summary><b>Click to expand — 20 tasks</b></summary> </details>

---

🟡 Moderate Performance (56–84%)

<details> <summary><b>Click to expand — 29 tasks</b></summary> </details>

---

❌ Weak Performance (≤ 55%)

<details> <summary><b>Click to expand — 18 tasks</b></summary> </details>

---

BLiMP Summary by Linguistic Category

Interpretation: Stentor2-12M shows solid grammatical intuitions for surface-level agreement, basic anaphora, and simple filler-gap dependencies — phenomena learnable from distributional patterns in text. It struggles most with phenomena requiring abstract syntactic sensitivity: scope-based NPI licensing, island extraction constraints, and complementizer selection under long-distance movement. These results are typical for a 12M parameter base model and broadly consistent with expectations from the scaling literature.

---

Training Dynamics

The training run processed 480,116,736 tokens across 2 epochs, stopping when the token budget was reached at approximately step 3,662 (well before the configured max_train_steps of 9,552).

Epoch 0 (~2,350 steps, 3,497s): Loss dropped rapidly from ~5.0 to ~3.2. Best checkpoint checkpoints updated at steps 1,500 (3.5410) and 2,400 (3.3743). Epoch-end eval: loss 3.3776, ppl 29.30.

Epoch 1 (~1,310 additional steps, 1,906s): Continued improvement. New best at step 3,300 (loss 3.2814). Token budget hit shortly after step 3,600. Epoch-end eval: loss 3.2565, ppl 25.96.

Throughput: ~91,600–92,000 tokens/sec average throughout training.

---

Use Cases & Intended Uses

---

Out-of-Scope Uses

• User-facing applications of any kind — No safety filtering, no alignment, no factual reliability.
• Medical, legal, or financial advice — 12M parameters cannot store or reason over specialized knowledge.
• Generating content about real people — Outputs mentioning real people are likely to be fabricated.
• Automated content pipelines — Output quality is insufficient for unreviewed publication.
• Non-English use — Vocabulary is English-only.
• Instruction following — This is a base model.

---

Ethical Considerations & Societal Impact

Inherited Data Biases

Trained on FineWeb-HQ and StenCore, a filtered subset of Common Crawls on the web and PDFs. Inherits:

• Western-centric perspective — Educational content skews toward Western viewpoints.
• English monolingualism — Training data and vocabulary are English-only.
• Demographic underrepresentation — Groups underrepresented in English educational web content will be underrepresented in outputs.

No Safety Tuning

No RLHF, no DPO, no constitutional AI, no content filtering.

Positive Aspects

• Democratizing AI research — Trained entirely on free-tier Kaggle compute.
• Transparency — Full training hyperparameters, architecture details, and deep details published.
• Minimal environmental footprint — ~2.1 hours of dual-GPU compute.

---

Inference Guide

Basic Generation

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained("StentorLabs/Stentor2-12M", torch_dtype=torch.float16)
tokenizer = AutoTokenizer.from_pretrained("StentorLabs/Stentor2-12M", trust_remote_code=True)
model = model.to("cuda").eval()

def generate(prompt, max_new_tokens=50, temperature=0.9, top_p=0.65):
    input_ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long).to(model.device)
    attention_mask = torch.ones_like(input_ids)
    with torch.inference_mode():
        output = model.generate(
            input_ids,
            attention_mask=attention_mask,
            max_new_tokens=max_new_tokens,
            do_sample=True,
            temperature=temperature,
            top_p=top_p,
            repetition_penalty=1.15,
            pad_token_id=tokenizer.pad_token_id,
        )
    new_ids = output[0][input_ids.shape[1]:].tolist()
    return tokenizer.decode(new_ids).strip()

print(generate("The history of computing began"))

---

🚀 Free Inference — Try It Now

No GPU, no setup, no API key required.

StentorLabs hosts a free, unlimited inference demo for all Stentor models at:

🔗 https://huggingface.co/spaces/StentorLabs/StentorLabs-demo_space

Features

All parameters (temperature, top-p, repetition penalty, max tokens) are fully customizable. Inference is free and unlimited.

---

Real Model Responses

The following are unfiltered outputs from Stentor2-12M. The first three are the very first generations run (no cherry-picking), the fourth is a deliberately bad example, and the fifth is a deliberately good one.

---

Sample 1 — temp=0.8, top_p=0.7, rep_penalty=1.2, max_tokens=140 Prompt: 5 Tips and Tricks for Difficult Homework.

SPECIAL STUDY SUBJECT: The Earth's Fall Times <22><22><22> Review the article with reference to
"A Tyre" <22><22><22> Revise your answers carefully and read it carefully. Write down what you
think you did and what you think it had to do with your answer. <22><22><22> Review your notes.
Read a book from the book, and review your answers. What would you do? <22><22><22> Do some
research findings on your own reading. <22><22><22> Invite students to read them. <22><22><22>
Try to learn the text by using different sources. <22>

The model completely ignores the prompt and defaults to academic/educational framing. PDF tokens (<22>) appear throughout as list separators — a characteristic artifact of the training data.

---

Sample 2 — temp=0.8, top_p=0.7, rep_penalty=1.3, max_tokens=140 Prompt: How to Properly Cite Sources

The Yellowstone Elementary School (CFP) is committed to providing an integrated and well-rounded
education system for students of all ages. The curriculum is designed to promote academic
achievement, instructional opportunities and a wider range of subject areas of study and practice
as well as teaching the student through its own learning style. This program is an important part
of the school's success in school and homework are set out in the district. We believe that all
pupils who receive a free school year will be able to attend classroom lessons from grade level
levels and will be required to complete their classes at any time during their summer break. We
are pleased that our current PSHCE

Coherent, grammatically sound institutional prose — but completely unrelated to citing sources. Classic Stentor2 behavior: latches onto educational register and produces confident, fluent non-sequitur.

---

Sample 3 — temp=0.6, top_p=0.75, rep_penalty=1.3, max_tokens=120 Prompt: How to Lose Weight Fast

The U.S. Department of Agriculture (USDA) is committed to providing the necessary services to
the economy and professions of urban development, as well as support for the protection of the
environment and the preservation of all living things within the community. The U.S. Government
provides a framework for developing this information in partnership with local organizations that
provide education, health care, and employment opportunities for all people affected by the
COVID-19 pandemic. We aim to ensure that the government's commitment to sustainable housing
through increased infrastructure and public transportation systems; therefore, we seek to

No connection to weight loss. Government/policy language — another common register for this model. Fluent, well-structured, completely off-topic.

---

Sample 4 (Hand-Picked Bad) — temp=0.8, top_p=0.5, rep_penalty=1.0, max_tokens=40 Prompt: Academics

<22><22><22><22><22><22><22><22><22><22><22><22><22><22><22><22><22><22><22><22><22><22><22>
<22><22><22><22><22><22><22><22><22><22><22><22><22><22>

Pure PDF token collapse. This is what happens with rep_penalty=1.0 (off), top_p=0.5 (too low), and a one-word prompt. All three conditions together push the model into its PDF-token attractor state. Do not use these settings.

---

Sample 5 (Hand-Picked Good) — temp=1.0, top_p=0.7, rep_penalty=1.4, max_tokens=45 Prompt: How to do Homework Assignments

Reasoning and Decision Making This Project is a great way to help students to succeed in school.
The project helps the student achieve better, the more they need to grow their faiths and to
support them in success as they become

Clean, readable, no PDF tokens, no repetition. Not exactly on-topic (pivot to "project" framing is typical), but this is what good Stentor2 output looks like: coherent educational prose, correct grammar, sensible continuation.

---

Quantization

FP16

model = AutoModelForCausalLM.from_pretrained("StentorLabs/Stentor2-12M", torch_dtype=torch.float16)
model = model.to("cuda")

Dynamic INT8 (CPU)

import torch
model_int8 = torch.quantization.quantize_dynamic(
    model.to("cpu"),
    {torch.nn.Linear},
    dtype=torch.qint8,
)

---

Format Conversion

Convert to GGUF (for llama.cpp)

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && pip install -r requirements.txt

huggingface-cli download StentorLabs/Stentor2-12M --local-dir stentor2-12m

python convert_hf_to_gguf.py stentor2-12m/ \
  --outfile stentor2-12m.gguf \
  --outtype f16

./llama-quantize stentor2-12m.gguf stentor2-12m-q4_0.gguf q4_0
./llama-cli -m stentor2-12m-q4_0.gguf -p "The science of" -n 50

Convert to ONNX

pip install optimum[exporters]
optimum-cli export onnx \
  --model StentorLabs/Stentor2-12M \
  --task text-generation-with-past \
  stentor2-12m-onnx/

---

Speculative Decoding

Stentor2-12M can serve as a fast draft model to accelerate inference from larger Llama-family target models.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

draft_model = AutoModelForCausalLM.from_pretrained(
    "StentorLabs/Stentor2-12M", torch_dtype=torch.float16
).to("cuda")

target_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-1B",
    torch_dtype=torch.float16,
    device_map="auto"
)
target_tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B")

inputs = target_tokenizer("Explain the concept of recursion", return_tensors="pt")
outputs = target_model.generate(
    **inputs,
    assistant_model=draft_model,
    do_sample=True,
    max_new_tokens=100
)
print(target_tokenizer.decode(outputs[0], skip_special_tokens=True))

Important caveat: Stentor2 uses a different vocabulary (8,064-token TokenMonster) than standard Llama models (32,000-token BPE). This vocabulary mismatch means the target model's acceptance rate may be lower than with a vocabulary-compatible draft model.

---

Bias, Risks & Limitations

• Prompt Relevance: Outputs frequently off-topic for complex prompts.
• Factual Accuracy: All factual claims should be treated as unreliable.
• Context Boundary: Hard limit of 1,024 tokens.
• English Bias: TokenMonster English vocabulary; other languages tokenize poorly.
• Training Data Bias: Inherits biases in English-language educational web and PDF text.
• Hallucination: May produce confident but fabricated content.
• No Alignment: No RLHF, no DPO, no constitutional training.
• Tokenizer Efficiency: 8K TokenMonster vocabulary produces more tokens per word than standard 32K BPE. This is expected and not a bug.
• Shared Tensor Warning: Removed shared tensor {'lm_head.weight'} is expected behavior from tied word embeddings. Safe to ignore.

---

Related Work

Comparable Sub-50M Models

Comparison caveats: Perplexity numbers are not directly comparable across models — different validation sets, vocabularies, and tokenizers all affect the number.

Related Research Papers

---

Environmental Impact

---

Citation

@misc{izumoto2026stentor2_12m,
  title        = {Stentor2-12M},
  author       = {Kai Izumoto},
  year         = {2026},
  publisher    = {StentorLabs},
  howpublished = {\url{https://huggingface.co/StentorLabs/Stentor2-12M}},
  note         = {12.3M parameter LlamaForCausalLM base model trained on
                  FineWeb-HQ and StenCore with a TokenMonster 8K vocabulary.
                  Trained on 2x Tesla T4 GPUs. Apache 2.0 license.}
}

---

Model Card Contact

Questions, benchmarks, or feedback: [email protected] or open a discussion.

---

<p align="center"> Made with ❤️ by <a href="https://huggingface.co/StentorLabs">StentorLabs</a><br> <i>Democratizing AI through accessible, efficient models</i> </p>