TaoNet-pico-A2-pretrain

TaoNet Pico A2 Pretrain

Model summary

TaoNet Pico A2 Pretrain is a pretrained TaoNet A2 variant. It was trained as an optimized model for perplexity-filter based dataset preprocessing and model benchmarking, not as a general-purpose assistant or instruction-following model. This model uses the TaoNet MLA + RoPE architecture. It was trained on TaoData dataset, built for high-quality language modeling in small models. The training objective was standard language modeling.

Generation

Transformers

import torch
from transformers import AutoModelForCausalLM

model_id = "Lobakkang/TaoNet-pico-A2-pretrain"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=dtype,
).to(device)
model.eval()

Sample inference

import sys
from pathlib import Path

import torch
from transformers import AutoModelForCausalLM

ROOT = Path(__file__).resolve().parent
SRC = ROOT / "src"
if str(SRC) not in sys.path:
    sys.path.insert(0, str(SRC))

from taoTrain.inference.loading import load_tokenizer

MODEL_ID = "Lobakkang/TaoNet-pico-A2-pretrain"
TOKENIZER_PATH = ROOT / "tokenizer" / "tokenizer.model"

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
dtype = torch.bfloat16 if device.type == "cuda" else torch.float32

tokenizer = load_tokenizer(TOKENIZER_PATH)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
    torch_dtype=dtype,
).to(device)
model.eval()

prompt = "Hello world"
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output_ids = model.generate(
        input_ids=input_ids,
        max_new_tokens=32,
        do_sample=True,
        temperature=0.7,
        top_p=0.95,
    )

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Benchmarks

These results are from the pretrained checkpoint:

Limitations

This checkpoint is best treated as a pretrained research model rather than a normal-purpose application model.

• It was trained for perplexity-filter style preprocessing and filtering workflows.
• It is not tuned for chat, instruction following, or tool use.
• Output quality is expected to vary outside the data and evaluation distribution used during pretraining.
• As with other small language models, it can produce plausible but incorrect text.
• The model should be validated before any downstream deployment or product use.

Training

Model

• Architecture: TaoNet / MLA + RoPE
• Tokenizer: SentencePiece
• Framework: TaoTrain
• Dataset: TaoData

Hardware

• GPU: 1 x RTX 5090

Software

• Training framework: TaoTrain

License

MIT License.

Citation

@software{taonet_pico_a2_pretrain,
  title={TaoNet Pico A2 Pretrain},
  author={Lobakkang},
  year={2026},
  url={https://huggingface.co/Lobakkang/TaoNet-pico-A2-pretrain}
}

If you use TaoTrain or TaoData in a writeup, cite the corresponding project references in addition to this checkpoint.