Downloads · 30 days
3
13% of all-time downloads
RichardErkhov/INSAIT-Institute_-_BgGPT-Gemma-2-2.6B-IT-v1.0-awq
INSAIT-Institute_-_BgGPT-Gemma-2-2.6B-IT-v1.0-awq is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
3
13% of all-time downloads
All-time downloads
24
Public
Parameters
2.6B
2.3 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors2.2 GB · 98%
How the weights are stored.
I322B · 77%
From the Hugging Face model README
Quantization made by Richard Erkhov.
BgGPT-Gemma-2-2.6B-IT-v1.0 - AWQ
library_name: transformers tags:

INSAIT introduces BgGPT-Gemma-2-2.6B-IT-v1.0, a state-of-the-art Bulgarian language model based on google/gemma-2-2b and google/gemma-2-2b-it.
BgGPT-Gemma-2-2.6B-IT-v1.0 is free to use and distributed under the Gemma Terms of Use.
This model was created by INSAIT, part of Sofia University St. Kliment Ohridski, in Sofia, Bulgaria.
The model was built on top of Google’s Gemma 2 2B open models. It was continuously pre-trained on around 100 billion tokens (85 billion in Bulgarian) using the Branch-and-Merge strategy INSAIT presented at EMNLP’24, allowing the model to gain outstanding Bulgarian cultural and linguistic capabilities while retaining its English performance. During the pre-training stage, we use various datasets, including Bulgarian web crawl data, freely available datasets such as Wikipedia, a range of specialized Bulgarian datasets sourced by the INSAIT Institute, and machine translations of popular English datasets. The model was then instruction-fine-tuned on a newly constructed Bulgarian instruction dataset created using real-world conversations. For more information check our blogpost.


We evaluate our models on a set of standard English benchmarks, a translated version of them in Bulgarian, as well as, Bulgarian specific benchmarks we collected:
These benchmarks test logical reasoning, mathematics, knowledge, language understanding and other skills of the models and are provided at https://github.com/insait-institute/lm-evaluation-harness-bg. The graphs above show the performance of BgGPT 2.6B compared to other small open language models such as Microsoft's Phi 3.5 and Alibaba's Qwen 2.5 3B. The BgGPT model not only surpasses them, but also retains English performance inherited from the original Google Gemma 2 models upon which it is based.
First install the latest version of the transformers library:
pip install -U 'transformers[torch]'
Then load the model in transformers:
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
"INSAIT-Institute/BgGPT-Gemma-2-2.6B-IT-v1.0",
torch_dtype=torch.bfloat16,
attn_implementation="eager",
device_map="auto",
)
For optimal performance, we recommend the following parameters for text generation, as we have extensively tested our model with them:
from transformers import GenerationConfig
generation_params = GenerationConfig(
max_new_tokens=2048, # Choose maximum generation tokens
temperature=0.1,
top_k=25,
top_p=1,
repetition_penalty=1.1,
eos_token_id=[1,107],
do_sample=True
)
In principle, increasing temperature should work adequately as well.
In order to leverage instruction fine-tuning, your prompt should begin with a beginning-of-sequence token <bos> and be formatted in the Gemma 2 chat template. <bos> should only be the first token in a chat sequence.
E.g.
<bos><start_of_turn>user
Кога е основан Софийският университет?<end_of_turn>
<start_of_turn>model
This format is also available as a chat template via the apply_chat_template() method:
tokenizer = AutoTokenizer.from_pretrained(
"INSAIT-Institute/BgGPT-Gemma-2-2.6B-IT-v1.0",
use_default_system_prompt=False,
)
messages = [
{"role": "user", "content": "Кога е основан Софийският университет?"},
]
input_ids = tokenizer.apply_chat_template(
messages,
return_tensors="pt",
add_generation_prompt=True,
return_dict=True
)
outputs = model.generate(
**input_ids,
generation_config=generation_params
)
print(tokenizer.decode(outputs[0]))
Important Note: Models based on Gemma 2 such as BgGPT-Gemma-2-2.6B-IT-v1.0 do not support flash attention. Using it results in degraded performance.
Example usage with vLLM:
from vllm import LLM, SamplingParams
from vllm.inputs import TokensPrompt
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
"INSAIT-Institute/BgGPT-Gemma-2-2.6B-IT-v1.0",
use_default_system_prompt=False,
)
sampling_params = SamplingParams(
max_tokens=2048,
temperature=0.1,
top_k=25,
top_p=1,
repetition_penalty=1.1,
stop_token_ids=[1, 107],
)
llm = LLM(
model="INSAIT-Institute/BgGPT-Gemma-2-2.6B-IT-v1.0",
dtype="bfloat16",
enforce_eager=True
)
messages = [
{"role": "user", "content": "Кога е основан Софийският университет?"},
]
formatted_prompt = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
input_ids = tokenizer(
formatted_prompt,
add_special_tokens=False
).input_ids
prompt = TokensPrompt(prompt_token_ids=input_ids)
output = llm.generate(
prompt,
sampling_params
)
generated_text = output[0].outputs[0].text
print(generated_text)
The model and instructions for usage in GGUF format are available at INSAIT-Institute/BgGPT-Gemma-2-2.6B-IT-v1.0-GGUF.
We welcome feedback from the community to help improve BgGPT. If you have suggestions, encounter any issues, or have ideas for improvements, please:
Your real-world usage and insights are valuable in helping us optimize the model's performance and behaviour for various use cases.