uk-gender-swapper-aya-101

🇺🇦 uk-gender-swapper-aya-101

This model is a PEFT-fine-tuned version of the aya-101 large language model, adapted for Ukrainian gender swapping. It is designed to generate gender-swapped versions of input sentences, making it useful for tasks involving gender bias mitigation, inclusive language generation, or synthetic dataset creation.

Model Type: Aya-101 (12.9B) + PEFT (LoRA)
Language: Ukrainian 🇺🇦
Task: Gender Swapping (Text-to-Text Generation)

---

💡 Use Cases

• Bias analysis and mitigation in NLP models
• Synthetic data generation for gender-balanced corpora
• Exploration of linguistic gender structures in Ukrainian
• Educational and research purposes related to gender and language

---

🧠 Model Details

• Base Model: CohereForAI/aya-101
• Fine-tuning Method: Parameter-Efficient Fine-Tuning (PEFT) with LoRA
• Training Data: A custom parallel dataset of Ukrainian sentences with gender-swapped equivalents (e.g., male-to-female and female-to-male subject transformations)
• Tokenizer: Same as Aya-101 (LLaMA tokenizer)

---

🔧 How to Use

You can use this model with the peft library and transformers. Here’s an example using the merged adapter:

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from peft import PeftModel, PeftConfig

adapter_repo_id = "linndfors/uk-sent-gender-swapper_aya-101"

config = PeftConfig.from_pretrained(adapter_repo_id)
base_model_id = config.base_model_name_or_path
tokenizer = AutoTokenizer.from_pretrained(base_model_id)

base_model = AutoModelForSeq2SeqLM.from_pretrained(base_model_id)

base_model.resize_token_embeddings(len(tokenizer))
inference_model = PeftModel.from_pretrained(base_model, adapter_repo_id)

prefix = "Перефразуй це речення, змінивши гендерні сутності на протилежні (чоловічий <-> жіночий): "

def generate_swapped_response(input_text, inference_model, tokenizer): 
    inputs = tokenizer(prefix + input_text, return_tensors="pt")

    with torch.no_grad():
        outputs = inference_model.generate(**inputs, max_new_tokens=100, temperature=0.1)

    generated = tokenizer.decode(outputs[0], skip_special_tokens=True)

    return generated

input = "Він працює викладачем в університеті."
gender-swapped_output = generate_swapped_response(input, inference_model, tokenizer)

---

📊 Example

Input:
Він працює викладачем в університеті.
Output:
Вона працює викладачкою в університеті.

---

🏗️ Training Details

• Adapter Type: LoRA (rank=32)
• PEFT Library: peft
• Hardware: Trained on Colab Pro+ (A100)
• Epochs: 15
• Batch Size: 4
• Optimizer: adamw_torch
• Learning Rate: 5e-5

---

📁 Files and Artifacts

• adapter_config.json – PEFT configuration
• adapter_model.bin – Fine-tuned LoRA weights
• README.md – Description
• tokenizer_config.json, tokenizer.model, etc. – Tokenizer files from the base model

---

⚠️ Limitations & Warnings

• May not generalize well to complex or idiomatic language.
• Designed specifically for binary gender swaps in Ukrainian.
• Not suitable for tasks requiring preservation of named entities in all cases.

---

📜 Citation

If you use this model in your work, please cite the original Aya-101 paper and consider crediting this repository:

@misc{linndfors2024ukgenderswapper,
  title={uk-gender-swapper-aya-101},
  author={Olha Nahurna},
  year={2024},
  howpublished={\url{https://huggingface.co/linndfors/uk-gender-swapper-aya-101}}
}

---

🙤 Questions or Feedback?

Feel free to open an issue or reach out via the Hugging Face discussion tab.