alloma-8B-Instruct

Model Description

This is the 8B parameter version of our Uzbek-optimized Llama series. Also, check out our other models:

• uzlm/alloma-1B-Instruct
• uzlm/alloma-3B-Instruct

---

Our uzlm/alloma-8B-Instruct model has been continually pretrained with context length of 4096 tokens, on 3.6B tokens (67% English, 33% Uzbek), then SFT fine-tuned. Our customized tokenizer averages 1.7 tokens per Uzbek word vs. ~3.5 in the original Llama models, meaning 2x faster inference and longer effective context length on Uzbek text.

Methodology: Efficient Vocabulary Adaptation for Uzbek

The primary motivation for our technical approach is to create a model with a more efficient tokenizer for the Uzbek language. This ensures both faster inference speeds and a longer effective context length when processing Uzbek text, as fewer tokens are needed to represent the same amount of information.

To avoid the prohibitive cost of training from scratch, we adapted the powerful meta-llama/Llama-3.1 base model using an in-place vocabulary replacement strategy. We identified less relevant non-ASCII tokens in the original vocabulary and replaced them with our custom Uzbek tokens. This was performed without altering the model's architecture or total vocabulary size, carefully merging the new Uzbek BPE rules while preserving the original English ones.

To give the new tokens a meaningful starting point for training, we initialized their embeddings using subtoken averaging. Each new Uzbek token was broken down by the original tokenizer, and its new embedding was created by averaging the embeddings of its subtokens. This method allowed for highly efficient continual pretraining on our bilingual dataset, resulting in a model fully optimized for Uzbek.

---

Benchmarks 1B, 3B

Benchmarks 8B

<!-- | **[Behbudiy Nemo 12B Uz](https://huggingface.co/behbudiy/Mistral-Nemo-Instruct-Uz)** | 0 | 0 | 0 | 0 | 0 | 0 | 0 | -->

The results show that our Uzbek-optimized models consistently outperform their base counterparts in translation benchmarks (BLEU and COMET) on the FLORES+ Uz-En / En-Uz evaluation datasets and sentiment analysis in Uzbek language. Also, on the MMLU benchmark, which measures general language understanding across multiple tasks in English, and News classification tasks, our Uzbek optimized model showed slight decline because of catastrophic forgetting of original English instruction following. (The official Llama model’s MMLU score may differ from our score due to our evaluation method. Refer to the links below to see evaluation details.)

We’re eager to see how these models will contribute to Uzbek open-source and be used by our Uzbek 🇺🇿 community. 🚀

How to use

The uzlm/alloma-8B-Instruct model can be used with transformers in the following way. We recommend preprocessing Uzbek input to replace apostrophe (') with sequence (APST) to achieve our model's lower tokenizer fertility.

Use with transformers

import re, torch
from transformers import AutoModelForCausalLM, AutoTokenizer
import langid

DEVICE   = torch.device("cuda" if torch.cuda.is_available() else "cpu")
DTYPE    = torch.bfloat16
MODEL_ID = "uzlm/alloma-8B-Instruct"
PATTERN  = r"[’‘‚‛ʻʼʽʾʿˈˊˋˌˍ'\']"

tok = AutoTokenizer.from_pretrained(MODEL_ID, use_fast=True)
tok.padding_side = "left"
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=DTYPE,
    device_map="auto"
)

EOT = "<|eot_id|>"
SYSTEM = (
    f"{tok.bos_token}<|start_header_id|>system<|end_header_id|>\n"
    "You are a helpful assistant<|eot_id|>"
)

def prompt(user: str) -> str:
    return (
        SYSTEM +
        "<|start_header_id|>user<|end_header_id|>\n" +
        f"{user}{EOT}" +
        "<|start_header_id|>assistant<|end_header_id|>"
    )

def generate(user: str, max_new: int = 256) -> str:
    lang, confidence = langid.classify(user)
    clean_text = re.sub(PATTERN, "APST", user) if lang != "en" else user
    enc = tok(prompt(clean_text), return_tensors="pt").to(DEVICE)
    out = model.generate(**enc,
                         max_new_tokens=max_new,
                         bos_token_id=tok.bos_token_id,
                         eos_token_id=tok.convert_tokens_to_ids(EOT),
                         pad_token_id=tok.pad_token_id,
                         do_sample=False)
    txt = tok.decode(out[0], skip_special_tokens=False)
    txt = txt.split("<|start_header_id|>assistant<|end_header_id|>", 1)[1]
    return txt.split(EOT, 1)[0].replace("APST", "'").strip()

print(generate("Menga Alisher Navoiy haqida aytib ber."))

Information on Evaluation Method

To evaluate on the translation task, we used FLORES+ Uz-En / En-Uz datasets. We used the following prompt to do zero-shot Uz-En evaluation both for the base model and Uzbek-optimized model (for En-Uz eval, we changed the positions of the words "English" and "Uzbek").

  prompt = f"Input: {clean_text} \n\nYour task is to accurately translate the given Uzbek text into English.\n"
        "Output only the English translation, without any additional comments.\n"
        "\nPlease translate the following Uzbek text into English."

To assess the model's ability in Uzbek sentiment analysis, we used the risqaliyevds/uzbek-sentiment-analysis dataset (refer to behbudiy/uzbek-sentiment-analysis dataset). We used the following prompt for the evaluation:

prompt = f'''Input: {clean_text} \n\nGiven the following text, determine the sentiment as either 'Positive' or 'Negative.' Respond with only the word 'Positive' or 'Negative' without any additional text or explanation."
'''

For Uzbek News Classification, we used risqaliyevds/uzbek-zero-shot-classification dataset and asked the model to predict the category of the news using the following prompt:

prompt = f'''Input: {clean_text}\n\nClassify the given news article in Uzbek.
0 - Siyosat - If the text is about politics.
1 - Iqtisodiyot - If the text is about the economy.
2 - Texnologiya - If the text is about technology.
3 - Sport - If the text is about sports.
4 - Madaniyat - If the text is about culture.
5 - Salomatlik - If the text is about health.
6 - Oila va Jamiyat - If the text is about family and society.
7 - TaAPSTlim - If the text is about education.
8 - Ekologiya - If the text is about ecology.
9 - Xorijiy Yangiliklar - If the text is about foreign news.

Print only one digit ID of the corresponding class.
'''

On MMLU, we performed 0-shot evaluation using the following template and extracted the first token generated by the model for measuring accuracy:

template = "Given the above question and choices, choose the single best answer (A, B, C, or D). Respond with only one letter..

Acknowledgements

This project was developed by the teams at Examy.me and Teamwork.uz. Their collaboration and resources were essential to the creation and success of the alloma model series.

More

For more details and examples, refer to the base model below: https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct