ANDREA-120M-ft

ANDREA-120M-ft

A 145.1M parameter decoder-only language model grown on a single RTX 4090 with a bandit-controlled curriculum over open data. Part of a permacomputer project: open source, open data, open weights.

Weights here are bit-identical to the CUDA training checkpoint step_852106.bin at step 852,106 — the export is verified tensor by tensor, not assumed.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("russellbal/ANDREA-120M-ft", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("russellbal/ANDREA-120M-ft", trust_remote_code=True)

ids = tok.apply_chat_template(
    [{"role": "user", "content": "What is a permacomputer?"}],
    add_generation_prompt=True, return_tensors="pt")
print(tok.decode(model.generate(ids, max_new_tokens=64, do_sample=True,
                                temperature=0.7)[0]))

trust_remote_code=True is required: ANDREA is its own architecture, and the three *_andrea.py files in this repo are its reference implementation.

Prompt format matters more than usual here

The corpus writes dialogue on a single line as > question / < answer, with turns joined by /. The chat template shipped with this repo produces exactly that, and staying inside it is the difference between coherent output and word salad — a bare prose prompt is out of distribution for this model.

Documents are also separated by BOS during training, and the tokenizer prepends it by default. Encoding with add_special_tokens=False drops it and starts the model mid-document, which measurably degrades generation. Sampling at temperature=0.7 matches how the model was sampled during training; greedy decoding at this scale degenerates into repetition.

An unconditional sample (input_ids=[[bos]], temperature=0.7) looks like:

> what is the most popular breed of dog? / < the most popular breed of dog is ...
> How does GPU architecture affect gaming performance? / < Modern GPU architectures
  like NVIDIA's Ada Lovelace and AMD's ...

Architecture

ANDREA is deliberately neither GPT-2 nor LLaMA. Reimplementing it means matching all of these:

• RMSNorm with no learnable gain. There is no norm weight in this checkpoint because the architecture has none.
• Learned absolute position embeddings. No RoPE, no ALiBi. The model cannot extrapolate past 1024 tokens and raises rather than guessing.
• A norm on the embedding sum, applied once before layer 0.
• A gainless final RMSNorm on the residual stream before the LM head. It is part of the architecture: serving without it puts the raw residual scale into the softmax and cross-entropy lands in the hundreds.
• ReLU MLP at 4x. Not GELU, not SwiGLU.
• No biases on any projection.
• Untied LM head — lm_head.weight is its own tensor, not wte reused.

x = wte[tok] + wpe[pos]
x = rmsnorm(x)
per layer:
    x = x + o_proj(attn(rmsnorm(x)))
    x = x + down_proj(relu(up_proj(rmsnorm(x))))
logits = lm_head(x)

RMSNorm is x * (mean(x^2) + 1e-05) ** -0.5.

Tokenizer

Harris (1955) successor-variety segmentation: morpheme boundaries discovered from the distributional properties of text, with no language-specific rules. The vocabulary is 256 raw byte fallbacks, then 24,576 learned segments, then BOS at id 24832. Any byte sequence encodes and round-trips, so text the segment table has never seen degrades to bytes rather than to an unknown token.

Training

Trained on the megachat-v8 corpus through a phase-based UCB1 bandit curriculum that reweights sources as the model's per-source loss moves. Data was filtered for dirty text, non-English, and assistant refusals before training.

Limitations

A model this size is a research artifact. It produces fluent English and has absorbed real factual material, but it hallucinates, has no alignment tuning, and has no safety filtering beyond the corpus cleaning described above. It was not built for and should not be used for advice, moderation, or any decision affecting a person.

Provenance

Full training history, probe analyses, and reproduction instructions live with the source: https://git.unturf.com. Companion papers: ANDREA-WHITEPAPER (training) and ANDREA-PROBES (15 interpretability probes).