magtina350m-base

MagTina350m — base

MagTina350m-base is the 354.6 M-parameter Brazilian-Portuguese foundation model trained from scratch by Dataseek under the ** Magestic.ai ** brand. This is the pretraining checkpoint — see dataseek/magtina350m-instruct for the instruction-tuned version.

Model summary

Note on logit_softcap. The original Mag350m model applied tanh(x/15)*15 to the output logits during training. To stay compatible with stock LlamaForCausalLM (and thus vLLM / TGI / transformers without trust_remote_code), this release drops the softcap. On 629 random positions the conversion produced 100 % top-1 token agreement with the original model in FP32. Effects on sampling-temperature behavior are negligible.

Training

Corpus mix (PT-BR only)

No private corpora, no proprietary subscriptions. Per-source dedup → cross-source dedup → quality filter → 17.39 B unique tokens.

Evaluation

200-example sample of each benchmark (matched protocol vs Tucano reference):

Lambada-PT (long-context coherence) and Calame-PT (cloze) are at-or-above Tucano-630m despite 1.8 × fewer parameters and ~half the pretrain tokens — credit to the v3 BPE tokenizer (-8.1 % total fertility vs Tucano) and 2 048-token training context. ARC-PT and BPB still trail.

Use

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("dataseek/magtina350m-base")
model = AutoModelForCausalLM.from_pretrained(
    "dataseek/magtina350m-base", torch_dtype=torch.float16).to("cuda")

prompt = "O Brasil é um país"
ids = tok(prompt, return_tensors="pt").input_ids.to("cuda")
out = model.generate(ids, max_new_tokens=80, do_sample=True,
                     temperature=0.8, top_p=0.9, repetition_penalty=1.1)
print(tok.decode(out[0], skip_special_tokens=True))

This is a completion model — no chat template, no special tokens needed at inference. For chat / assistant use, switch to dataseek/magtina350m-instruct.

Intended use & limitations

Intended use. Research, derivative fine-tunes, PT-BR language-modeling baselines.

Out of scope. Production deployment without further alignment, non-Portuguese tasks, factual question-answering requiring up-to-date or specialised knowledge.

Limitations.

• 354 M params is small — expect frequent factual errors, weak multi-step reasoning, and brittle code/math.
• PT-BR only — minimal exposure to English (~1 % of pretrain), zero exposure to other languages.
• Knowledge cutoff: early 2026.
• Public-data only; biases of CommonCrawl, Wikipedia PT, and PT-BR news media are present and unaudited.

Citation

@misc{magtina350m2026,
  author = {Frasson, Ricardo and {Dataseek Team}},
  title  = {MagTina350m: A 354 M-parameter Brazilian Portuguese language model},
  year   = {2026},
  publisher = {Hugging Face},
  url    = {https://huggingface.co/dataseek/magtina350m-base}
}

License

CC-BY-NC 4.0 — free for research and non-commercial derivative work; commercial use requires written permission from Dataseek (contact via dataseek.com.br).