Downloads · 30 days
8
7% of all-time downloads
kdrnyzv890/qwen2.5-3b-karakalpak-base
qwen2.5-3b-karakalpak-base is a machine learning model from kdrnyzv890. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
This repository contains a LoRA adapter for Qwen2.5-3B, specifically fine-tuned on a Karakalpak language corpus.
Downloads · 30 days
8
7% of all-time downloads
All-time downloads
120
Public
Repo size
193 MB
Likes
2
Public
Click a slice to open those files.
.safetensors120 MB · 91%
From the Hugging Face model README
This repository contains a LoRA adapter for Qwen2.5-3B, specifically fine-tuned on a Karakalpak language corpus.
This is a Continued Pre-training (CPT) adapter.
Training on this project was paused at Checkpoint 3000 due to a lack of computational budget. As an independent developer working on a low-resource language, the costs of GPU power have outpaced my personal resources. To reach a full 1.0+ Epoch and proceed to the Instruction Tuning (Chat) phase, I am seeking support from the community.
How you can help:
q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj).You must load this adapter on top of the base Qwen/Qwen2.5-3B model.
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch
model_id = "Qwen/Qwen2.5-3B"
adapter_id = "kdrnyzv890/qwen2.5-3b-karakalpak-base"
# Load Tokenizer and Base Model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")
# Load the Karakalpak Adapter
model = PeftModel.from_pretrained(model, adapter_id)
# Example: Text Completion
text = "Nókis qalası - Qaraqalpaqstan Respublikasınıń"
inputs = tokenizer(text, return_tensors="pt").to("cuda")
with torch.no_grad():
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))