Gemma9B_Inkuba_LoRa_adapter_v1

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

This model card contains a LoRa adapter for the Gemma-9b-it model. It aims to improve Machine Translation performance into South African langauges.

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

The LoRa adapter was trained on a subset of the lelapa/Inkuba-instruct dataset targeting the langauges English, isiZulu and isiXhosa. The data was sampled from the Inkuba-instruct dataset using langauge-ids.

Model Sources [optional]

• Developed by: Sandil Bhikha
• Model type: LoRa adapter
• Language(s) (NLP): English, isiZulu, isiXhosa
• Finetuned from model: Gemma-9b-it

<!-- Provide the basic links for the model. -->

• Paper [optional]: [More Information Needed]
• Demo [optional]: [More Information Needed]

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Aims to improve Machine Translation performance on South African langauges. Based on Gemma-9b-it, so the model is generally task adaptable.

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

The datatset was sampled from lelapa/Inkuba-instruct dataset. Minor inconsistencies were present in translations from the training data.

How to Get Started with the Model

Use the code below to get started with the model.

base_model_id = "google/gemma-2-9b-it"
adapter_model_id = "sbhikha/Gemma9B_Inkuba_double_quant"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)

double_quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    quantization_config = double_quant_config,
    device_map="auto"
)

config = PeftConfig.from_pretrained(adapter_model_id)
peft_model = PeftModel.from_pretrained(model, adapter_model_id)

text = "Translate this from English to isiZulu: Sawubona, unjani?"

input_ids = tokenizer(text), return_tensors="pt").to("cuda:0")

outputs = peft_model.generate(**input_ids, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

English

• sbhikha/Inkuba_english_train_sliced_v1
• sbhikha/Inkuba_english_dev_sliced_v1

Zulu

• sbhikha/Inkuba_isizulu_train_sliced_v1
• sbhikha/Inkuba_isizulu_dev_sliced_v1

Xhosa

• sbhikha/Inkuba_xhosa_train_sliced_v1
• sbhikha/Inkuba_xhosa_dev_sliced_v1

Training Hyperparameters

• Training regime: bf16 mixed precision <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->

LoRa Config:

• Rank = 64
• Alpha = 32
• Target Modules (Linear Modules): ['q_proj', 'gate_proj', 'up_proj', 'k_proj', 'v_proj', 'o_proj', 'down_proj']
• Dropout = 0.05

Training Arguments:

• Learning Rate: 2e-4
• per_device_train_batch_size: 1
• gradient_accumulation_steps: 4
• warmup_steps = 3
• max_steps = 100
• optim="paged_adamw_8bit"

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: 1x H100 GPU
• Hours used: 23.7
• Cloud Provider: Lambda Labs
• Compute Region: Utah, USA
• Carbon Emitted: [More Information Needed]