tiyat_alpha_arch1

Tiyat Alpha

Para leer esta ficha en español, haz clic aquí.

Tiyat is a 184.1M-parameter multilingual foundation language model trained from scratch with majority representation of Indigenous languages of Mexico. The training corpus includes most of Mexico's 68 national Indigenous languages and their approximately 360 variants. To the best of our knowledge, Tiyat is the first LLM developed specifically for this linguistic diversity.

Unlike an adaptation of a pretrained model centered on English or another high-resource language, Tiyat arch1 started from random weights. Its tokenizer, training mixture, and decoder-only Transformer were built as part of the Tachiwin research pipeline. Indigenous languages account for a target 70% of the training mixture rather than being included as a marginal addition.

Intended use and expected behavior

Tiyat arch1 is a base autoregressive language model for research on multilingual representations, tokenization, and language modeling with low-resource languages at the center of the training distribution. It is suitable for short-form text continuation, qualitative generation experiments, reproducible education, and study of a small LLM trained entirely from scratch.

The model continues a prompt by predicting the next token. It is not instruction-tuned or optimized for chat, translation, or factual question answering. Prompts containing several words generally provide better signals about the intended language, register, and topic than short or ambiguous inputs.

Generation quality varies across languages and variants according to the available training data. Outputs may mix languages, lose coherence, or become repetitive, particularly in longer generations. The model has not yet undergone comprehensive evaluation for every language and variant, so generated text should be reviewed by a knowledgeable speaker when linguistic accuracy matters.

Architecture

Tiyat arch1 is a custom Transformer decoder-only causal language model implemented with JAX and Flax/NNX. It is not a GPT-2, Llama, or other existing model variant.

The architecture uses learned token and positional embeddings, pre-normalized causal multi-head attention, GELU feed-forward layers, residual connections, dropout, a final layer normalization, and an independent output projection.

This repository includes both the original NPZ training artifact and a Safetensors conversion. For a Safetensors-only distribution, see tachiwin/tiyat_alpha_arch1_safetensors. Both repositories represent the same arch1 model and architecture.

Training data

• Dataset: tachiwin/tiyat-ground-pretrain-m1024
• Tokenizer: tachiwin/tokenizer_64k
• Approximate training volume: 300M tokens

The target training mixture was:

The corpus is multilingual and uneven: not every language or variant has the same volume, quality, domain coverage, or degree of review. Consult the dataset card for data-specific documentation and licensing conditions.

Training configuration

The final loss is a training statistic, not a cross-language quality score. The model has fewer than two training tokens per parameter and is primarily data-limited.

Usage

Install the runtime dependencies:

pip install jax "flax>=0.10" safetensors transformers huggingface_hub

The repository ships custom configuration and modeling code. Review that code before enabling trust_remote_code=True.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "tachiwin/tiyat_alpha_arch1"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
)

prompt = "Había una vez un gato que"
input_ids = tokenizer(prompt, return_tensors="np")["input_ids"][0].tolist()

output_ids = model.generate(
    input_ids,
    max_new_tokens=40,
    temperature=0.7,
    top_k=40,
    top_p=0.9,
    repetition_penalty=1.5,
    no_repeat_ngram_size=3,
    eos_token_id=tokenizer.eos_token_id,
    seed=0,
)

print(tokenizer.decode(output_ids, skip_special_tokens=True))

model.generate() is a minimal implementation included with this repository. It supports greedy decoding or temperature sampling with top-k, top-p, repetition penalty, and no-repeat n-gram blocking. It is not the complete Transformers GenerationMixin: beam search and GenerationConfig are not currently supported.

Open research and technological sovereignty

Tiyat is part of Tachiwin's work toward technology that communities can study, reproduce, modify, and shape—not merely access as a closed external service. The project documents the broader pipeline, including document digitization, corpus construction, tokenization, architecture, training, and inference.

• Technical and project article
• Reproducible training and inference tutorial
• Public demo on Hugging Face
• Tachiwin

License

The model repository is released under the Apache License 2.0. Linked datasets, source documents, and other resources may have their own licenses or conditions; consult their respective repositories before reuse.

About Tachiwin

Tachiwin is an Indigenous, Mexican, community-led, nonprofit, open-source initiative developing AI and language technology for the Indigenous languages of Mexico. Its horizon is technology capable of identifying, translating, and conversing across the country's 68 national Indigenous languages and their variants, guided by technological sovereignty, collaboration, non-extractivism, and shared knowledge.