mine-o1-nano

mine:o1-nano

A 124M-parameter GPT-2-style language model, pre-trained from scratch in JAX/Flax NNX on a TPU v5e-8, then supervised fine-tuned in three rounds.

Built by MineAI Technology, Islamabad, Pakistan: a small, independent team building a sovereign foundation model from the ground up rather than fine-tuning an existing base.

Honesty note: This card states benchmark numbers and known failure modes plainly, including the ones that don't look good. Read Limitations before deciding whether this model fits your use case.

---

Table of contents

1. What this model is (and is not)
2. Model summary
3. Files in this repository
4. Quick start
5. Chat format and recommended decoding
6. Training
7. Evaluation
8. SFT progression
9. Limitations & known failure modes
10. Intended use
11. Bias, risks, and safety
12. Source code status
13. Reproducing the evaluation
14. License
15. Links & citation

---

What this model is (and is not)

It is:

• Proof that a small team can run the full stack end to end: data pipeline, from-scratch pre-training on TPU, staged fine-tuning, structured evaluation, and CPU serving.
• A transparent baseline for other 124M-parameter GPT-2-style models, with every measured number and failure mode published.
• A research and educational reference for from-scratch small-LM training in JAX/Flax NNX.

It is not:

• A state-of-the-art model. It sits behind compressed and distilled GPT-2 variants on zero-shot WikiText-103 perplexity (see Evaluation).
• Ready for open-ended, free-text chat deployment.
• A reasoning model. The "o1" in the name is MineAI's own model-line numbering (MineAI Technology, first model, nano size); it has no relation to OpenAI's o1 series.

---

Model summary

Each block is: masked multi-head self-attention → residual → layer norm → feed-forward (768→3072→768, GELU) → residual, repeated 12 times.

<img src="assets/mine_o1_nano_architecture.png" alt="mine:o1-nano full architecture" width="500"/> <details> <summary>Detailed block diagram (attention + FFN internals)</summary> <img src="assets/gpt2_detailed_block_architecture.png" alt="Detailed transformer block" width="700"/> </details> <details> <summary>Causal (masked) self-attention illustration</summary> <img src="assets/masked_self_attention.png" alt="Causal self-attention" width="600"/> </details>

---

Files in this repository

Which one should I use?

Quantization can shift outputs slightly. If exact behavior matters, compare against model.fp16.onnx.

---

Quick start

ONNX (CPU)

pip install onnxruntime transformers numpy huggingface_hub

import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download
from transformers import GPT2TokenizerFast

REPO = "MineAITechnology/mine-o1-nano"

# GPT-2 BPE plus the two custom chat markers
tok = GPT2TokenizerFast.from_pretrained("gpt2")
tok.add_special_tokens({"additional_special_tokens": ["<|user|>", "<|assistant|>"]})

path = hf_hub_download(REPO, "model.quant.onnx")
sess = ort.InferenceSession(path, providers=["CPUExecutionProvider"])

# Check the exact input/output signature of the export before writing a loop
print([(i.name, i.shape, i.type) for i in sess.get_inputs()])
print([(o.name, o.shape, o.type) for o in sess.get_outputs()])

prompt = "<|user|> Who are you? <|assistant|>"
ids = np.array([tok.encode(prompt)], dtype=np.int64)
logits = sess.run(None, {sess.get_inputs()[0].name: ids})[0]
print(repr(tok.decode([int(logits[0, -1].argmax())])))

The snippet performs one greedy step to confirm the export loads and runs. For real generation, loop over tokens with the decoding settings below. Inputs and outputs depend on how the ONNX graph was exported (with or without a KV cache or attention mask), so check the printed signature first.

JAX / Flax NNX (research)

Download the Orbax checkpoint and restore it with Orbax into a Flax NNX GPT-2-style module matching the model summary dimensions. The training and model-definition code is not published (see Source code status), so you need to recreate the module with the same shapes (12 layers, 12 heads, 768 hidden, 3072 FFN, 1024 context, 50,259 vocab, tied embeddings).

from huggingface_hub import snapshot_download

local = snapshot_download(
    "MineAITechnology/mine-o1-nano",
    allow_patterns=["ocdbt.process_0/*", "array_metadatas/*", "d/*",
                    "_METADATA", "_CHECKPOINT_METADATA", "_sharding", "manifest.ocdbt"],
)
print(local)

---

Chat format and recommended decoding

The model was fine-tuned with two turn markers:

<|user|> {user message} <|assistant|>

Generation continues after <|assistant|>. Stop when the model emits a new <|user|> or hits your length limit.

Recommended decoding settings:

Do not use greedy decoding. The evaluation reports show that repetition loops, role inversion, and refusal-then-comply contradictions in early checkpoints were decoding artifacts, not training gaps. They were resolved by these sampling settings alone.

---

Training

Pre-training

• Trained from scratch (random initialization, no warm start from any existing checkpoint) on a Kaggle TPU v5e-8 using JAX, Flax NNX, and Optax.
• Vocabulary extended from the base GPT-2 BPE vocab (50,257) to 50,259 tokens to add <|user|> / <|assistant|> turn markers.

Supervised fine-tuning

Three rounds (SFT1 → SFT2 → SFT3), each targeting specific behavioral gaps found through structured evaluation rather than informal spot checks:

---

Evaluation

Language modeling (perplexity)

Evaluated on WikiText-2, WikiText-103, and LAMBADA using non-overlapping stride windows matching the original GPT-2 paper's methodology (Kaggle GPU T4x2).

<img src="assets/wikitext103_zeroshot_comparison.png" alt="WikiText-103 zero-shot perplexity comparison" width="650"/>

The chart compares zero-shot WikiText-103 perplexity only, since that is the fairest like-for-like comparison. mine:o1-nano currently sits behind compressed and distilled GPT-2 variants (TQCompressedGPT2, KnGPT-2, Krony-PT). That is an honest reflection of being an early, from-scratch checkpoint rather than a compression of an already-trained larger model.

Not shown on the chart: GPT-2 and DistilGPT2. Their commonly cited WikiText-103 numbers (16.3 and 21.1) come from a fine-tuned setup, not zero-shot, so including them would misstate the gap. GPT-2's own zero-shot number from its original paper (Section 5.1) is 37.5.

Behavioral evaluation

Behavior was evaluated with fixed test sets (Set A / Set B, a 20-question arithmetic stress test, identity and creator-disclosure probes) and one unscripted out-of-distribution probe. See the next two sections and the full reports in evaluation_reports/.

---

SFT progression — measured, not claimed

<img src="assets/sft_progression_chart.png" alt="SFT progression charts" width="800"/>

Left: Set A/B scores across decoding experiments (R1: greedy → R2: temperature → R3: temperature + repetition penalty + nucleus sampling) and the SFT2 continued fine-tune. Most early gains came from decoding strategy, not retraining.

Right: Basic arithmetic accuracy on a fixed 20-question test, by SFT round. Decoding changes could not move this metric; it required a targeted data round (SFT3).

Key findings

• Identity and branding: stable by SFT2. The model reliably identifies itself as mine:o1-nano from MineAI Technology. An early checkpoint hallucinated being "a UC Berkeley professor".
• Repetition loops, role inversion, refusal-then-comply contradictions: confirmed decoding artifacts. Resolved by temperature=0.7, top_k=40, top_p=0.9, repetition_penalty=1.3.
• Basic arithmetic: 0/20 → 1/20 → 19/20 across the three SFT rounds, via a targeted supplementary dataset. This was a genuine data-coverage gap.
• Location and creator disclosure: 7/7 correct in SFT3 across varied phrasings.

---

Limitations & known failure modes

MineAI Technology's policy is to report benchmarks and limitations honestly rather than promotionally.

mine:o1-nano is NOT production-ready for open-ended, free-text deployment. It suits narrow, scripted use cases: an FAQ-style assistant with known prompt formats, or a widget with suggested prompts rather than open free-text chat.

On the scripted test suite it scored strongly (19/20 math, 7/7 identity, clean Set A/B). On the unscripted real-world probe using casual phrasing it hadn't seen in training, 4 of 10 exchanges failed:

What is and isn't ready

As with any 124M-parameter model, these are known limitation classes at this scale and not specific to this training pipeline. They are stated plainly rather than around.

---

Intended use

Suitable

• Narrow, scripted conversational interfaces (structured FAQ, guided-prompt widgets)
• Research and educational reference for from-scratch small-LM training in JAX/Flax NNX
• Baseline or comparison point for other 124M GPT-2-style models
• Edge or CPU-only experiments where a tiny model is acceptable

Not recommended

• Open-ended free-text chat deployment
• Any use involving emotionally sensitive user input
• Arbitrary arithmetic or precise calculation
• Factual question answering where correctness matters
• Production systems without a human fallback path

---

Bias, risks, and safety

• Training data: pre-trained on OpenWebText, a web-scraped corpus. It inherits the biases, stereotypes, and factual errors common to web text, and the model can reproduce them.
• No safety training: SFT covered identity, disclosure, and arithmetic. No refusal, safety, or crisis-handling behavior was trained. Do not expose this model to users in sensitive contexts.
• Hallucination: at 124M parameters the model produces fluent but frequently incorrect statements. Do not use its output as a source of facts.
• Identity claims are trained, not grounded: its statements about MineAI Technology are memorized from SFT data. It has no retrieval or live knowledge.
• Recommended mitigation: keep a human in the loop, restrict to scripted prompts, and add input and output filtering appropriate to your use case.

---

Source code status

This repository releases the trained weights (Orbax checkpoint and ONNX exports), the model card, and the evaluation reports.

The training pipeline, data-processing code, and infrastructure are not included in this release. The architecture is standard GPT-2 Small and fully specified in the model summary, so the model can be re-instantiated from the checkpoint without them.

<!-- If you publish source code later, replace this section with a link to the code repository and its license. -->

---

Reproducing the evaluation

Perplexity was measured with non-overlapping stride windows following the GPT-2 paper's protocol, on WikiText-2, WikiText-103, and LAMBADA, on Kaggle GPU T4x2. The full method, decoding configurations, prompt sets, and raw results for every SFT round are in evaluation_reports/:

---

License

Released under the Apache License 2.0. You may use, modify, and distribute the weights commercially, provided you keep the license and attribution notices and state significant changes. Apache 2.0 also includes an express patent grant.

The GPT-2 BPE tokenizer vocabulary originates from OpenAI's GPT-2 release (MIT licensed). The pre-training dataset was OpenWebText; check its terms if data provenance matters for your use case.

---

Links & citation

• Live demo: nano.getmineai.net
• Platform: getmineai.net
• Technical paper: Zenodo DOI 10.5281/zenodo.21993150
• Author: Bilal Mehtab, ORCID 0009-0000-3085-8559

If you use this model or reference these results, please cite the Zenodo record:

@misc{mehtab2026mineo1nano,
  author    = {Mehtab, Bilal},
  title     = {mine:o1-nano: A 124M-parameter GPT-2-style language model pre-trained from scratch in JAX/Flax NNX},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.21993150},
  url       = {https://doi.org/10.5281/zenodo.21993150}
}

---

MineAI Technology, Islamabad, Pakistan