Gemma-2-9b-translator

Model Card for GEMMA-2 Post-Processing Model

This model is based on the GEMMA-2 architecture and is fine-tuned specifically to improve YouTube auto-generated captions. It post-processes the captions by correcting misaligned words that occur due to phonetic similarities but do not fit the context. The model is designed to enhance the quality of speech-to-text output, making the captions more accurate and readable by identifying and fixing contextually incorrect words with similar pronunciations.

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

• Developed by: SangWoo Han
• Model type: Causal Language Model (AutoModelForCausalLM)
• Language(s) (NLP): English
• Finetuned from model [optional]: google/gemma-2-9b-it

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Direct Use

This model can be directly applied to improve the quality of YouTube auto-generated captions by post-processing them. Without any additional fine-tuning, the model can correct phonetically similar words that are incorrectly transcribed but out of context, resulting in more accurate and contextually appropriate captions. Users can input raw captions generated from YouTube videos, and the model will output refined, corrected text.

Out-of-Scope Use

The model is not designed for general-purpose language correction, grammar checking, or tasks unrelated to phonetic error correction in captions. It is not suitable for handling highly technical or specialized language that it has not been trained on, nor for correcting non-phonetically related errors in transcriptions. Misuse could include attempts to employ the model for purposes such as translation, sentiment analysis, or plagiarism detection, which are beyond its intended scope.

Bias, Risks, and Limitations

The model may exhibit biases inherent in the dataset used during training, including but not limited to regional accents, linguistic variations, and dialects. The correction capability may also perform unevenly across different languages or types of content. Additionally, the model may struggle with extremely noisy audio or highly specialized terminology not present in the training data.

Recommendations

It is recommended that users be aware of potential biases toward certain accents or dialects. Testing the model on a diverse range of speech inputs can help evaluate its effectiveness across various demographics. In cases where specific speech patterns or accents are frequent, further fine-tuning may be required to mitigate these biases. Users should also remain mindful of the limitations in correcting technical jargon or highly specialized content that the model has not been trained on.

How to Get Started with the Model

Use the code below to get started with the model. https://colab.research.google.com/drive/1AJ9s_wZNXxvco79J_X_s-U60OqC3Fa02?usp=sharing

from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    AutoModelForSeq2SeqLM,
    BitsAndBytesConfig
)
import torch
from peft import PeftModel, PeftConfig

peft_model_id = "navgod/Gemma-2-9b-translator"
config = PeftConfig.from_pretrained(peft_model_id)
model = AutoModelForCausalLM.from_pretrained(peft_model_id)
model = PeftModel.from_pretrained(model, peft_model_id)
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)

model = model.to("cuda")

Training Details

Training Data

https://huggingface.co/datasets/jesscusi/english-slang

Training Procedure

from peft import LoraConfig

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.2,
    bias="none",
    task_type="CAUSAL_LM",
)

training_arguments = TrainingArguments(
    output_dir=new_model,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=1,
    gradient_accumulation_steps=2,
    optim="adamw_torch",
    num_train_epochs=1,
    eval_strategy="steps",
    eval_steps=0.2,
    logging_steps=1,
    warmup_steps=10,
    logging_strategy="steps",
    learning_rate=2e-4,
    fp16=False,
    bf16=False,
    group_by_length=True,
    report_to="wandb"
)

Training Hyperparameters

• Training regime: [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Testing Data

<!-- This should link to a Dataset Card if possible. -->

[More Information Needed]

Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

[More Information Needed]

Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

[More Information Needed]

Results

[More Information Needed]

Summary

Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: [More Information Needed]
• Hours used: [More Information Needed]
• Cloud Provider: [More Information Needed]
• Compute Region: [More Information Needed]
• Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

[More Information Needed]

Compute Infrastructure

[More Information Needed]

Hardware

[More Information Needed]

Software

[More Information Needed]

Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

BibTeX:

[More Information Needed]

APA:

[More Information Needed]

Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->

[More Information Needed]

More Information [optional]

[More Information Needed]

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

[More Information Needed]