tinctura-v1

tinctura-v1

A 96.2M parameter decoder-only language model pretrained from scratch on 75B tokens. It is the cagliostro-v3 recipe at two thirds of the size: the same architecture, data, schedule and token count, with 18 layers instead of 30.

On the Open SLM Leaderboard Index it scores 20.81. Among models under 100M parameters that places it second, 0.26 behind Rose-1.5-Medium at 21.07.

The training run is complete. 75.00B tokens, 762,939 steps, learning rate decayed to zero.

<video controls src="https://huggingface.co/bench-labs/tinctura-v1/resolve/main/tinctura-v1.mp4" width="100%"></video>

Results

Zero-shot, measured on the public weights in this repository with exactly the commands under Reproducing the evaluation: the lm-evaluation-harness hf backend and the leaderboard's official ArithMark-3 script, both in float32.

The Index is the leaderboard's own formula, (N(HellaSwag,25) + N(CombinedARC,25) + N(PIQA,50) + 0.65*N(ArithMark,25)) / 3.65 where N(v,c) = 100(v-c)/(100-c) and CombinedARC is the mean of ARC-Easy and ARC-Challenge. Everything further down that tracks training uses the training-time evaluation harness. That harness tokenizes each answer separately from its question and ran ArithMark-3 in bfloat16, so it reads a little differently: 20.77 for the final checkpoint, with individual benchmarks up to 0.7 points apart from the table above. The numbers in this table are the ones to compare against the leaderboard.

The top of the sub-100M field, using the leaderboard's published figures. tinctura-v1 is scored with the same formula but is not yet listed on the board.

Against the leader the result is split. tinctura-v1 is ahead on ARC-Easy and PIQA, level on HellaSwag, and behind on ARC-Challenge and ArithMark-3. ArithMark-3 is most of the gap: 2.3 points there cost about 0.55 Index at its 0.65 weight, more than the whole 0.26 margin. Rose-1.5-Medium was trained on roughly 100B tokens according to its card, against 75B here.

What a third fewer parameters costs

Because tinctura-v1 follows cagliostro-v3 step for step, the two runs can be compared at identical points in training. Several of these comparisons were run side by side on the same machine with the same evaluation code, using checkpoints from both repositories. Every number in this section comes from the training-time harness, for both models.

Through the stable phase the smaller model trailed by about 1.7 Index over the first 20B tokens and by about 2.9 after that. Single evaluations carry roughly one point of noise, so individual gaps ranged from 0.5 to 3.9. Just before the cooldown, at step 646,000, the paired evaluation read 19.34 against 21.96.

The cooldown is where the two models part. cagliostro-v3 gained 4.6 Index over the last 15% of training. tinctura-v1 gained 1.4. The shortfall came in two places. From step 646,000 to 706,000 cagliostro-v3 added 1.95 and tinctura-v1 only 0.21. From 706,000 to 736,000 the two moved together, 1.32 and 1.23. Over the final stretch to the end cagliostro-v3 added another 1.32 while tinctura-v1 held flat. On ARC-Challenge and ArithMark-3 the smaller model finished the cooldown slightly lower than it started it.

After the first few billion tokens the training loss gap to cagliostro-v3 settled between 0.07 and 0.10 and stayed there through the cooldown. The sharp drop at 63.75B is the data mixture changing, not the model improving.

The cooldown was run twice

The first attempt at the cooldown hit a bug in the data loader. When training switches to the cooldown mixture, each process restarts its data-loading threads. Threads blocked on a full queue could miss the stop signal and keep producing batches from the old mixture next to the new ones, so each process trained on a blend. A standalone test of the loader leaked old-mixture batches in 9 of 10 trials, 20 to 44% of batches after the switch. The loss told on it: at the switch the loss fell by 0.26 where cagliostro-v3's fell by 0.49.

The loader was fixed so that every restart gets its own queue and old threads exit on their own, and the fixed version leaked nothing in the same test. The cooldown was then rerun from the last checkpoint before it, step 644,000. The rerun's loss fell by 0.45 at the switch and tracked cagliostro-v3 as expected. The weights in this repository come from the rerun.

The fix did not change the benchmarks in a measurable way. At step 706,000 the first attempt scored 19.68 and the rerun 19.55, which is within evaluation noise. The data-mixing bug was real, but it was not what held the cooldown gain down.

Model details

The architecture is defined in this repository and is the same code as cagliostro-v3. trust_remote_code=True is required because CagliostroForCausalLM is not part of transformers.

Training data

The same mixtures and shards as cagliostro-v3. The first mixture covers the stable phase, and the second takes over when the cooldown begins at 85% of the run.

Mathematics rises from 10% to 28% during the cooldown.

Training setup

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "bench-labs/tinctura-v1"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype=torch.float32)

ids = tok("The capital of France is", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=32, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

This is a base model with no instruction tuning and no chat template. It completes text.

Samples

Twelve fixed prompts, one sample each, unedited. Sampling at temperature 0.7, top-p 0.9, repetition penalty 1.1, 48 new tokens, seed 0 before every prompt. These are the same prompts used for cagliostro-v3. The film shows the first sentence of three of them. The misses are left in.

<details> <summary>All 12 samples</summary>

1. The water cycle begins when

The water cycle begins when the oceans absorb more carbon dioxide than they release. As a result, they are rapidly cooling and ocean currents move more quickly.
“This is a pretty good analogy for how climate change might happen,” said Professor Charles G. Hines,

2. Photosynthesis is the process by which

Photosynthesis is the process by which plants convert light energy into chemical energy. The reaction produces oxygen and water as a by-product.
What is photosynthesis?
Photosynthesis is a type of photosynthetic process that occurs in plants. Photosynthesis takes place in the roots

3. To solve 47 + 68, first

To solve 47 + 68, first add the digits together:
\[ 4 + 7 + 8 = 11 + 8 = 19 \]

2. Now subtract the sum from 5 (the number we want to find) from

4. The Roman Empire fell because

The Roman Empire fell because of the threat of the Black Death. The Romans were not interested in fighting off a disease, but rather had an idealism for the destruction of their enemies and the preservation of civilization. This idea was reflected in the policies of Rome, which

5. A prime number is a number that

A prime number is a number that has exactly two distinct positive divisors: 1 and itself.

The sum of the digits of an integer is equal to the sum of its digits multiplied by the number of digits in its base-$d$ representation, where

6. The three states of matter are

The three states of matter are also called the solids, liquids and gases.
- Atoms are found in a single layer called an atom.
- The number of atoms is inversely proportional to the energy.
- A gas has molecules that have very small amounts

7. Gravity is the force that

Gravity is the force that holds everything in place, while gravity is what pulls everything towards the center of the universe.

Now, let's talk about energy. Energy is a form of power, and it can be transformed from one form to another through various processes

8. In 1969, astronauts

In 1969, astronauts from the International Space Station (ISS) were to use their space suits for a brief period of time before being sent back to Earth. The astronauts also wore a variety of spacesuits and space suits, which they used to protect them from extreme

9. The heart pumps blood through

The heart pumps blood through the body and then the arteries deliver the oxygen to the tissues.
- Blood is made up of white and red blood cells, which carry oxygen from the lungs to all parts of the body.
- The red blood cells are also known

10. To find the area of a rectangle you

To find the area of a rectangle you need to know two things: the length and width.

The area of a rectangle is given by the formula A = L * W. In this case, we know that L = 10 and W = 12.

11. Volcanoes form when

Volcanoes form when the surface of a volcano cools and becomes very hot. The resulting heat causes a lot of steam to rise up through the volcano's vents, which are areas where magma is released. When this steam comes out of the vents

12. The difference between weather and climate is

The difference between weather and climate is more important than what the term suggests. The temperature in a given location can be found by measuring how much it gets warmer or colder than the average of that region. If you have a map of your local area, you can look at its

</details>

Reproducing the evaluation

pip install lm-eval
python -m lm_eval --model hf \
  --model_args pretrained=bench-labs/tinctura-v1,dtype=float32,trust_remote_code=True \
  --tasks hellaswag,arc_easy,arc_challenge,piqa \
  --num_fewshot 0 --batch_size 8 --device cuda:0

ArithMark-3 uses the leaderboard's official script from AxiomicLabs/ArithMark-3.0. The script defaults to bfloat16, so pass --dtype float32:

python bencharithmark-3.py --model bench-labs/tinctura-v1 --data-path arithmark-3.jsonl --device cuda --dtype float32

Evaluate in float32 throughout. The model uses a logit cap of 15, and in bfloat16 the ArithMark-3 score moved between 37.7 and 38.5 depending on the GPU. The exported weights agree with the training checkpoint exactly.

Provenance

This repository holds the training history. A checkpoint was pushed about every 30 minutes, and any of them can be retrieved by revision. The history starts at step 22,000, because pushes during the first half hour failed on a repository permission error.

The checkpoints from the first cooldown attempt described above were removed from this history, so every revision after step 644,000 belongs to the rerun that produced the published weights.

Limitations

English only. 2,048 token context. No instruction tuning, no safety tuning, no RLHF. At 96M parameters it confabulates freely, as the samples show, and should not be relied on for factual questions. The mathematics ability measured by ArithMark is arithmetic and short symbolic work, not general mathematical reasoning.

License

Apache-2.0. The training data is drawn from FineWeb-Edu and FineMath (ODC-By), DCLM-Baseline and OpenMathInstruct-2 (CC-BY-4.0), Cosmopedia v2 and SmolTalk (Apache-2.0).