Downloads · 30 days
0
budaya-lab/Bntng-disambiguation-SeaLion-v1
Bntng-disambiguation-SeaLion-v1 is a machine learning model from budaya-lab. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
This model performs word sense disambiguation for Jawi script text, specifically addressing cases where the same written form can correspond to multiple different words in Malay. The model is fine-tuned from SEA-LION…
Downloads · 30 days
0
Access
Public
Updated Nov 6, 2024
Repo size
671 MB
Likes
0
Public
Click a slice to open those files.
.safetensors336 MB · 100%
From the Hugging Face model README
This model performs word sense disambiguation for Jawi script text, specifically addressing cases where the same written form can correspond to multiple different words in Malay. The model is fine-tuned from SEA-LION (Southeast Asian Language Intelligence Open Network), specifically the 7B instruction-tuned research variant.
Example of ambiguous forms handled:
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
base_model_path = "aisingapore/sea-lion-7b"
tokenizer = AutoTokenizer.from_pretrained(base_model_path, trust_remote_code=True)
peft_model_path = "mevsg/bntng-dis-v1"
# Load the base model and tokenizer
base_model = AutoModelForCausalLM.from_pretrained(base_model_path)
# Load the PEFT finetuned model on top of the base model
model = PeftModel.from_pretrained(base_model, peft_model_path)
# Example usage
prompt = "Apakah perkataan yang sesuai menggantikan 'bntng' dalam ayat berikut:"
input_text = "Langit malam ini penuh dengan bntng yang bersinar terang."
full_prompt = f"### USER:\n{prompt} {input_text}\n\n### RESPONSE:\n"
# Tokenize the prompt and create the attention mask
tokens = tokenizer(full_prompt, return_tensors="pt", padding=True)
# Generate the output while passing the attention_mask
output = model.generate(
tokens["input_ids"],
attention_mask=tokens["attention_mask"],
max_new_tokens=20,
eos_token_id=tokenizer.eos_token_id
)
# Decode the generated tokens to text
generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)