HyGPT-10b

HyGPT-10b

HyGPT-10b is the first Armenian large language model that has been pretrained on corpus of Armenian text data. This model is designed to understand and generate Armenian text, making it a pioneering high-quality language model specifically created for the Armenian language.

Model Details

Model Description

HyGPT-10b is a decoder-only language model based on Google's Gemma-2-9b architecture that has been further pretrained on 10B tokens of Armenian text.

A key technical modification in this model is the decoupling of the embedding and LM head layers, allowing the output layer to be trained independently, which can improve the model's ability to generate accurate Armenian text.

• Developed by: Gen2B & NCCAIT
• Model type: Decoder-only language model
• Language(s) (NLP): Armenian, English, Russian
• Technical Report: link
• License: HyGPT Permissive Use License

Uses

First, install the Transformers library with:

pip install -U transformers

Then, run this example:

from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig
import torch

model_path = "Gen2B/HyGPT-10b"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.float16,
    device_map="auto",
)

PROMPT = 'Ինչու է խոտը Կանաչ:'

inputs = tokenizer(
    PROMPT,
    return_tensors="pt",
)

print("Generating...")
generation_output = model.generate(
    input_ids=inputs["input_ids"].cuda(),
    generation_config=GenerationConfig(
        temperature=0.0001,
        repetition_penalty=1.1,
        do_sample=True
    ),
    return_dict_in_generate=True,
    output_scores=True,
    max_new_tokens=256,
)
for s in generation_output.sequences:
    print(tokenizer.decode(s))

# Կանաչ գույնի առկայությունը բուսականության մեջ պայմանավորված է կլորավուն քլորոֆիլային մոլեկուլների առկայությամբ, որոնք հանդիսանում են լույսի անդրադարձման և տարածման միակ աղբյուրները։ Առողջ բույսերում այդպիսի մոլեկուլներ շատ են, ուստի դրանցից արտացոլվող լույսի երանգը համապատասխանում է կանաչին։ Եթե ​​մոլեկուլների թիվը նվազում է, օրինակ՝ սովի կամ վիրուսային վարակի դեպքում, ապա բույսերի գույնը փոխվում է. Դեղին-շագանակագույն, կարմիր, կապույտ, սև։

Direct Use

HyGPT-10b can be used directly for:

• Armenian text generation
• Question answering in Armenian
• Text completion for Armenian content
• Understanding Armenian language queries

Bias, Risks, and Limitations

• The model may reflect biases present in the Armenian language training data
• Accuracy may vary across different Armenian dialects and regional variations
• The model may not have up-to-date knowledge beyond its training data
• Like all language models, it may occasionally generate incorrect or nonsensical responses
• The model's understanding of specialized Armenian terminology may be limited in certain domains

Training Details

Training Data

The model was pretrained on a diverse corpus of Armenian text data comprising approximately 10 billion tokens. The dataset includes:

• Armenian web content
• Armenian literature and publications
• Armenian news articles
• Armenian Wikipedia
• Other publicly available Armenian text sources

The data was collected with a focus on representing Armenian language usage across various domains and topics.

Preprocessing

The Armenian text data underwent several preprocessing steps:

• Cleaning and normalization of Armenian text
• Removal of duplicate content
• Tokenization using the base Gemma tokenizer

Training Procedure

The model was further pretrained from the google/gemma-2-9b base model using a pretraining approach. A key modification was decoupling the embedding and LM head layers, allowing the output layer to be trained independently. This approach was adopted based on a series of short experiments followed by evaluation on three publicly available Armenian language datasets. The results demonstrated that training the embedding and output layers separately yields higher accuracy according to metrics, compared to both the standard synchronized training of embedding and output layers, as well as configurations with frozen embedding layer, frozen output layer, or both layers frozen. The table below shows the evaluation results across different configurations:

Results

The model demonstrates strong performance on Armenian language tasks, showing significant improvements over models without Armenian-specific pretraining. Detailed benchmark results will be published in the future.

Summary

HyGPT-10b shows promising capabilities for Armenian language understanding and generation, making it a valuable resource for Armenian NLP applications. Additionally, the model serves as an excellent foundation model for further fine-tuning on specific data and domains, allowing developers to adapt it to specialized Armenian language tasks and industry-specific applications.

---

License and Terms of Use

This model is based on Gemma and is distributed according to the Gemma Terms of Use.

Notice: Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms.

Modifications Notice

This model is a modified version of the original Gemma-2-9b model. The modifications include:

1. Further pretraining on 10 billion tokens of Armenian text data
2. Decoupling of the embedding and LM head layers to allow independent training of the output layer

Use Restrictions

According to the Gemma Terms of Use, the model should not be used:

1. For purposes outlined in the Gemma Prohibited Use Policy
2. In violation of applicable laws and regulations

Disclaimer of Warranty

UNLESS REQUIRED BY APPLICABLE LAW, THE GEMMA SERVICES, AND OUTPUTS, ARE PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING ANY WARRANTIES OR CONDITIONS OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. YOU ARE SOLELY RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING, REPRODUCING, MODIFYING, PERFORMING, DISPLAYING OR DISTRIBUTING ANY OF THE GEMMA SERVICES OR OUTPUTS AND ASSUME ANY AND ALL RISKS ASSOCIATED WITH YOUR USE OR DISTRIBUTION OF ANY OF THE GEMMA SERVICES OR OUTPUTS AND YOUR EXERCISE OF RIGHTS AND PERMISSIONS UNDER THIS AGREEMENT.