meta-llama-Llama-3.2-7B

<p align="left"> <img src="./toksuite-logo.png" alt="TokSuite Logo" width="260"/> </p>

TokSuite – Llama-3.2 - 7B parameters

Model Summary

TokSuite–Llama-3.2-7B is part of TokSuite, a suite of language models designed to study the impact of tokenizer choice on language model behavior under controlled conditions.

This model uses the Llama-3.2 tokenizer and is otherwise identical to the other TokSuite-Llama models (toksuite/meta-llama-Llama-3.2-1B and toksuite/meta-llama-Llama-3.2-300M) training data, training budget, embedding parameters, data order and only differ in architecture (non-embedding parameters).

---

Tokenizer

• Tokenizer: Llama-3.2
• Tokenization method: BPE
• Vocabulary size: 128,256
• Out-of-vocabulary handling: Byte-fallback
• Language coverage: Multilingual
• Pretokenization source: GPT-4

Processing details:

• Numbers: Group by 3
• Contractions: GPT-4
• Unicode normalization: None
• Whitespace / boundary markers: Learned
• Zerowidth chars: Token

Why Llama-3.2?

Llama-3.2 was included in TokSuite to represent a multilingual BPE tokenizer with modern GPT-4–style pretokenization and a medium-to-large vocabulary size. As described in the tokenizer selection rationale of the TokSuite paper, Llama-3.2 exemplifies a contemporary tokenizer design that combines subword segmentation with preprocessing conventions used in recent large language models.

Including Llama-3.2 enables TokSuite to study tokenizer behavior in settings where:

• BPE segmentation is paired with GPT-4–style pretokenization,
• vocabulary size is substantially larger than early English-centric tokenizers,
• and multilingual text is handled through a single shared tokenizer.

This makes Llama-3.2 a representative example of modern multilingual BPE tokenization.

---

Model Architecture

• Architecture: Decoder-only Transformer (Lingua's Llama-3.2-7B configuration)
• Non-embedding parameters: ~7B
• Context length: 4096 tokens
• Framework: Meta Lingua
• Initialization: Shared super-vocabulary initialization across TokSuite models

The architecture and training setup are identical across all TokSuite models; only the tokenizer differs.

---

Training Data

The model was trained on a multilingual corpus totaling approximately 100B tokens, composed of:

• English: 40B tokens from FineWeb-Edu
• Multilingual: 60B tokens evenly distributed across:
  • Chinese (ZH)
  • Turkish (TR)
  • Italian (IT)
  • Farsi (FA)

You can find the pretraining dataset here: toksuite/toksuite_pretraining_data

All TokSuite models are trained using a fixed token budget, following common practice in large-scale language model training.

---

Training Procedure

• Training steps: 100,000
• Sequence length: 4096
• Batch size: 256 sequences
• Optimizer: AdamW
• Peak learning rate: 1e-3
• Learning rate schedule: Cosine decay with 2,000 warm-up steps
• Weight decay: 0.1

---

TokSuite Robustness Benchmark

TokSuite–Llama-3.2 variants are evaluated on the TokSuite robustness benchmark, which measures sensitivity to real-world text perturbations, including:

• orthographic and spelling variations,
• diacritics presence and absence,
• keyboard and input-method noise,
• Unicode formatting and homoglyphs,
• OCR and spacing artifacts,
• LaTeX and STEM-style formatting.

Tokenization Robustness under Multilingual Text Perturbations
Values represent relative performance drop, computed as (Acc_clean − Acc_perturbed) / Acc_clean, where lower values indicate greater robustness.

Perturbation types include:

• Input: non-native keyboard input and romanization
• Diacr.: optional diacritics
• Orth.& Gram.: orthographic and grammatical errors
• Morph: morphological variations including derivations, inflections, and contractions
• Noise: homoglyph substitutions, OCR artifacts, typos, and spacing errors
• LaTeX: LaTeX-style mathematical formatting
• STEM: scientific diagrams and notational conventions
• Unic.: Unicode styling characters

NEN denotes non-English inputs and EN denotes English inputs. The Avg column reports the average relative performance drop across all perturbation categories.

---

Intended Use

This model is intended for:

• research on tokenization and robustness,
• multilingual NLP analysis,
• controlled ablation studies,
• benchmarking tokenizer behavior under noise.

It is not instruction-tuned, aligned, or optimized for deployment.

---

Limitations

• Trained on a limited set of five languages.
• Not optimized for instruction following or dialogue.
• Fixed token budget constrains exposure to raw text depending on tokenization efficiency.
• Intended strictly for research purposes.

---

Ethical Considerations

TokSuite models are released to support scientific investigation of tokenization effects.
They may reflect biases present in large-scale web data and should not be used in high-stakes or user-facing applications without additional safeguards.

---

Citation

If you use this model, please cite:

@article{toksuite2025,
  title={TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior},
  author={Alt{\i}nta\c{s}, G\"ul Sena and Ehghaghi, Malikeh and Lester, Brian and Liu, Fengyuan and Zhao, Wanru and Ciccone, Marco and Raffel, Colin},
  year={2025},
  arxiv={https://arxiv.org/abs/2512.20757},
}