vwen-1.5B-instruct

Model Card for Model ID

LoRA SFT model Qwen2-1.5B on 600k sample of OpenHermes vietnamese (translated)

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

• Developed by: thangvip
• Model type: Qwen2
• Language(s) (NLP): Vietnamese
• Finetuned from model: Qwen/Qwen2-1.5B

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

Have not trained on RLHF for safety

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("thangvip/vwen-1.5B-instruct", device_map="auto", cache_dir="./cache").eval()
tokenizer = AutoTokenizer.from_pretrained("thangvip/vwen-1.5B-instruct", cache_dir="./cache")

messages = [
  {'role': 'system', 'content': "bạn là trợ lý AI hữu ích"},
  {"role": "user", "content": "Nước nào có diện tích lớn nhất?"},
]
text = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = tokenizer(text, return_tensors="pt")
inputs = {k: v.to("cuda") for k, v in inputs.items()}

outputs = model.generate(**inputs, tokenizer=tokenizer, max_new_tokens=256, do_sample=True, top_p=0.95, temperature=0.1, repetition_penalty=1.2, eos_token_id=tokenizer.eos_token_id, stop_strings=['<|im_end|>'])
print(tokenizer.decode(outputs[0]))

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

Trained on OpenHermes dataset (translated to Vietnamese) > 600k samples

Training Hyperparameters

<!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->

• Target_modules: q_proj, k_proj, v_proj, o_proj, up_proj, down_proj, gate_proj
• batch_size: 2048
• epoch: 1
• optimizer: paged_adamw_8bit
• warmup_ratio: 0.02
• learning_rate: 1e-5