Minueza-3-95M-Base

Minueza-3-95M-Base

Summary

Minueza-3-95M-Base is a compact English language model with the Gemma 3 architecture, trained from scratch on 1.95 billion tokens. It declares a context length of 8192, which is what the sliding-window layout supports, but every training step ran at 2048 tokens; treat anything past that as extrapolation. At 94.7 million parameters it is a lightweight foundation meant to be fine-tuned, not used as-is.

It is distributed only as GGUF, because GGUF was the only weight format used during training: there is no PyTorch checkpoint behind these files. The f32 file is the training master and can continue pretraining; the q8_0 and q4_0 files are for inference.

Being this small, the model has severe limitations in reasoning and factual knowledge, and will produce fluent text that is frequently wrong. It was trained on web text and can reproduce the biases in it.

Usage

With llama.cpp:

llama-completion -hf Felladrin/Minueza-3-95M-Base:Q8_0 -c 8192 -n 256 \
  --temp 0.7 --top-p 0.85 --top-k 30 --min-p 0.02 \
  --repeat-penalty 1.15 --repeat-last-n 128 \
  -no-cnv -p "This book tells the story"

-hf downloads and caches the weights for you. Swap :Q8_0 for :Q4_0 to pull the smaller copy, or -m <file>.gguf to point at a local one.

This is a base model: it completes text and has no chat template. The repetition penalty above is not optional, a model this size loops without it.

The q4_0 file is 55 MB, small enough to run in a browser tab through Wllama.

Intended Uses

• A base for fine-tunes: instruct, classification/routing, reranking. Minueza-3-95M-RP is one, with its corpus mix and hyperparameters published.
• Running fast on machines without a GPU, and inside browsers via Wllama.
• A from-scratch training artifact to study, with the optimizer state published so the pretraining can be continued rather than restarted.

Fine-tunes

That card publishes the corpus mix and the hyperparameters of both stages, though not the corpus itself, so it is a worked example of fine-tuning this base with gguf-trainer and never leaving GGUF rather than a recipe you can re-run end to end. Its numbers come from the same four-task battery as the second table under Evaluation below and compare directly against it; they do not compare against the llama-perplexity tables above it, which use a different harness and, on ARC, a different split. Open a discussion here if you train one and want it listed.

Model Architecture

Gemma 3, with sliding-window attention on 5 of every 6 layers.

Special Tokens

Eleven tokens were reserved before pretraining and are frozen into the vocabulary, so a later fine-tune can use them without touching the embedding matrix. Pretraining emitted exactly one of them, <|endoftext|>, between documents. The other ten never appeared as a target. Their rows are not frozen (embeddings are tied here, so every row is a softmax competitor on every step and takes gradient), but nothing ever taught them a meaning, and the first fine-tune whose data uses a token is what gives it one. Nothing can be added later, because the vocabulary and the embedding matrix both froze at step 1, so a fine-tune that needs a marker has to reuse one of these or spell it out in plain text.

The <|...|> tokens are CONTROL, so llama.cpp handles them and never renders them as visible text. The <...> tokens are USER_DEFINED, so they survive into the output, where llama.cpp's --jinja reasoning and tool-call parsers can see them.

The set is ChatML plus Qwen-style reasoning and tool calling, which is what it was chosen for, but nothing in the weights commits you to that template. Any format works; only these eleven strings tokenize as a single atomic token, and everything else costs one token per piece.

EOS moves when you add a chat template. This base declares eos_token_id = 32757 (<|endoftext|>), the document separator it was trained on. A ChatML fine-tune ends turns with <|im_end|>, so its GGUF has to declare 32759 instead, or generation will not stop on a turn boundary. Everything else about the tokenizer stays identical.

A template has to render in your trainer as well as in llama.cpp. Training text and inference text must be byte-identical, so the same Jinja source has to work in both engines, and they are not equally permissive: llama.cpp uses minja, while the trainer used here uses @huggingface/jinja 0.3.3. A template that renders only under minja will throw when you tokenize your fine-tuning data, or worse, quietly diverge from it. Measured against 0.3.3, these constructs fail and need a rewrite:

These work: namespace(), macro (including default arguments), messages[::-1], | items, | tojson, raise_exception() and inline if expressions.

Suggested chat template

Nothing in the weights requires this one, but it is a working starting point: it uses all ten non-<|endoftext|> tokens, avoids every construct in the table above, and was checked to render byte-identically under @huggingface/jinja 0.3.3 and llama.cpp's minja across plain chat, a system prompt with multiple turns, reasoning in the history, and a tool call followed by a tool response. Note that chat-corpus renders its own built-in ChatML and takes no template flag, so training on this one means rendering the corpus yourself.

{%- if tools -%}
    {{- "<|im_start|>system\n" -}}
    {%- if messages[0]["role"] == "system" -%}
        {{- messages[0]["content"] + "\n\n" -}}
    {%- endif -%}
    {{- "<tools>\n" -}}
    {%- for tool in tools -%}
        {{- (tool | tojson) + "\n" -}}
    {%- endfor -%}
    {{- "</tools><|im_end|>\n" -}}
{%- elif messages[0]["role"] == "system" -%}
    {{- "<|im_start|>system\n" + messages[0]["content"] + "<|im_end|>\n" -}}
{%- endif -%}
{%- for message in messages -%}
    {%- if message["role"] == "tool" -%}
        {{- "<|im_start|>user\n<tool_response>\n" + message["content"] + "\n</tool_response><|im_end|>\n" -}}
    {%- elif not loop.first or message["role"] != "system" -%}
        {{- "<|im_start|>" + message["role"] + "\n" -}}
        {%- if message["reasoning_content"] -%}
            {{- "<think>\n" + message["reasoning_content"] + "\n</think>\n\n" -}}
        {%- endif -%}
        {{- message["content"] -}}
        {%- if message["tool_calls"] -%}
            {%- for call in message["tool_calls"] -%}
                {{- ("\n" if message["content"] or not loop.first else "") + "<tool_call>\n" + (call["function"] | tojson) + "\n</tool_call>" -}}
            {%- endfor -%}
        {%- endif -%}
        {{- "<|im_end|>\n" -}}
    {%- endif -%}
{%- endfor -%}
{%- if add_generation_prompt -%}
    {{- "<|im_start|>assistant\n" -}}
{%- endif -%}

With tools, a call and a response, that renders to:

<|im_start|>system
You are terse.

<tools>
{"type": "function", "function": {"name": "get_weather"}}
</tools><|im_end|>
<|im_start|>user
weather in Lisbon?<|im_end|>
<|im_start|>assistant
<tool_call>
{"name": "get_weather", "arguments": {"city": "Lisbon"}}
</tool_call><|im_end|>
<|im_start|>user
<tool_response>
18C
</tool_response><|im_end|>

Three details are load-bearing, in case you rewrite it. not loop.first or message["role"] != "system" keeps a system message at index 0 from being emitted twice while still rendering one that appears mid-conversation, rather than dropping it silently. The {%- if message["tool_calls"] -%} guard is not optional: @huggingface/jinja throws Expected iterable type in for loop on a missing key, where minja ignores it. And the inline ("\n" if message["content"] or not loop.first else "") is what stops an empty-content tool call from opening with a stray blank line.

Keep tool_calls[].function.arguments an object in your training data, not a JSON string. Given a string, the trainer renders "arguments": "{\"city\":\"Lisbon\"}" while llama.cpp parses it first and renders "arguments": {"city": "Lisbon"}. That is a silent divergence between the text you train on and the text you serve, and it was the only disagreement between the two engines. As objects they match exactly.

It leaves out two things on purpose. There is no default system prompt, because a base model's fine-tuner should choose it. And there is no enable_thinking or reasoning_effort switch, because both need the default filter; <think> still renders whenever a message carries reasoning_content.

Training

Two phases, both f32, both on a single AMD Strix Halo APU through the WebGPU backend.

Final training loss: 2.45 on the phase A blend, 2.78 on fineweb-edu at the end of phase B. The two are not comparable to each other, different corpora.

Training used Felladrin's GGUF Trainer, a from-scratch trainer written in TypeScript that keeps weights in GGUF end to end and runs its kernels on WebGPU, with no PyTorch anywhere in the stack. It is open source, and the two files below are exactly what it needs to pick this run up where it stopped:

hf download Felladrin/Minueza-3-95M-Base --local-dir base/
deno run -A cli.ts inspect --model base/Minueza-3-95M-Base.F32.gguf   # prints the flags to repeat
deno run -A cli.ts pretrain --data your.tokens --out out/continued.gguf \
  --resume base/Minueza-3-95M-Base.F32.gguf \
  --hidden 640 --layers 12 --head-dim 64 --window 1024 --max-seq 8192 \
  --steps 5000 --seq-len 2048 --batch 8 --lr 0.01

Keep the .optstate sidecar next to the F32 file and the optimizer resumes warm.

Two things that are easy to get wrong. --steps 5000 is a fresh WSD cycle, 500 steps of warmup and 1000 of cooldown, not a continuation of the 119,000 this run already did; pass --start-step as well if you want the original schedule carried on. And the vocab has to be this one, because the embedding matrix froze when pretraining started. tokenize has no flag for it, but it reuses a <out>.tokenizer.json that is already there, so copy the published file to your.tokenizer.json before running tokenize --out your. In the other order you get a freshly trained vocab in your.tokens and a sidecar describing a different one, which nothing downstream checks: every id then means the wrong string against the frozen embedding rows.

Evaluation

llama-perplexity --multiple-choice, HellaSwag validation (10,042 tasks) and ARC-Challenge validation, every model scored by the same script on the same machine.

Three other models went through the same harness as reference points:

Random baselines are 25.0 on both benchmarks. Read the numbers honestly: this model improves on its predecessors and loses to every reference on HellaSwag, including Supra2-100M-Base, which is the same size and 6.06 points ahead on 15x the training tokens. That gap is the whole story of this checkpoint: 1.95B tokens over 94.7M parameters is about 21 tokens per parameter, and the models above it all bought their lead with more of them, not with a different recipe. ARC-Challenge separates only SmolLM2-135M; every other model here, references included, sits inside noise of chance on it. The value of this checkpoint is as a starting point, not as a scorer.

The same weights also went through the four-task battery of the Open SLM Leaderboard, scored 0-shot on full sets with eval-choice from gguf-trainer, which matches lm-eval-harness on the HellaSwag query construction and on normalizing acc_norm by character length. Chance is 50.0 on PIQA and 25.0 on the other three.

These numbers are not comparable to the tables above, and the reference models were not scored this way. HellaSwag reads 29.25 up there and 30.14 here on identical weights and identical items, which is scoring method alone; ARC-Challenge additionally changes split, 299 validation items above against the full 1,172-item test set here. Compare within one table, never across.

The Index normalizes each task against its own chance level, averages the two ARC splits into one term before that, and divides the sum of PIQA, ARC, HellaSwag and 0.65 x ArithMark-3 by 3.65. It is self-computed, not a submitted entry, and ArithMark-3 is not implemented in eval-choice, so it enters the sum at chance; dropping the term and dividing by 3 instead gives 12.98. One sigma on a single index value at these set sizes is 0.71, which is worth keeping in mind before reading any small gap between checkpoints as real. Minueza-3-95M-RP, two further training stages on these weights, scores 10.77 on this same battery: a difference well inside that error.

Samples

Greedy decoding (--temp 0 --top-k 1) with the repetition penalty from the Usage section, which is a logit transform and leaves the run deterministic. -ngl 0 pins it to the CPU: repeat runs on one backend are byte-identical, but a GPU build can pick the other token at a near-tie and diverge from there, so the backend is part of the recipe.

llama-completion -hf Felladrin/Minueza-3-95M-Base:Q8_0 -c 512 -n 60 \
  --temp 0 --top-k 1 --repeat-penalty 1.15 --repeat-last-n 128 --seed 42 \
  -ngl 0 -no-cnv -p "<prompt>"

The capital of France is the city of Paris. The capital was founded in 1621 by King Louis XIV, who had been a French aristocrat and a great-grandson of King Charles I. In 1789, the French government established a new capital at Saint-Louis-sur-Ric

Photosynthesis is the process by which plants produce oxygen and carbon dioxide. The photosynthetic process involves the production of light, water and other substances from sunlight. The photosynthetic process can be divided into two phases: - Phase 1 - the first phase involves the conversion of light energy to chemical energy (photons

The best way to learn a new language is by practicing it. - Learn the basics of grammar and usage, including how to use them correctly. - Practice using your native tongue in order to improve your English speaking skills. - Use the correct pronunciation for each word you are learning.

• Listen carefully to what others have said

Once upon a time, there was a little robot who had been playing with the ball. The robot was called "the ball" and it was named after the famous American inventor, Charles Babbage. He invented the first computer that could read text in any language. The idea of using computers to solve problems was born from his childhood.

That is the honest shape of a 95M base model: fluent English, plausible structure, and facts that are mostly invented. Paris was not founded in 1621, Louis XIV was not alive then, and Charles Babbage was neither American nor a namer of robots. Drop the repetition penalty and greedy decoding collapses into loops within a sentence or two.

Quantization

All three files measured on the same machine: HellaSwag validation (10,042 tasks) through llama-perplexity --multiple-choice, and perplexity on the wikitext-2 raw test set at ctx 2048.

Q8_0 is indistinguishable from F32. The perplexity gap is 0.0106 against an error bar of +/- 0.2429, and the HellaSwag gap is 0.08 against +/- 0.45; both sit inside the noise, and greedy decoding from the two files produces identical tokens on most prompts. Use q8_0 for inference; the f32 file is worth its 380 MB only as a training master.

Q4_0 splits the two metrics, and the split is the useful part. It costs 7.4% perplexity while leaving HellaSwag untouched. Perplexity is sensitive to the probability mass on every token, so 4-bit rounding shows up at once; multiple choice only needs the right continuation to rank first, and that ordering survives the added noise. So q4_0 degrades free-running generation and is a worse starting point for a fine-tune, while remaining fine for ranking, scoring, and classification, which is what the browser-sized copy is mostly for.

Caveat on reading any of these: at ~29 against a 25.0 floor, HellaSwag has about four points of headroom here, so it cannot resolve small quality differences. Perplexity is the sensitive instrument at this scale.

Files

Limitations

• Facts are unreliable. It writes fluent sentences about things that are not true.
• English only. The vocab is English-centric, so another language needs a new tokenizer and a fresh pretrain.
• No chat template and no instruction following. That is a fine-tune's job.
• It loops without a repetition penalty.

License

Apache License 2.0.