lab21-llama3.2-3b-r8

LoRA Adapter (Rank 8) - Llama 3.2 3B Vietnamese

LoRA adapter (rank=8) fine-tuned on Vietnamese Alpaca dataset

Model Details

• Base Model: unsloth/Llama-3.2-3B-Instruct-bnb-4bit
• LoRA Rank: 8
• LoRA Alpha: 16
• Target Modules: q_proj, v_proj
• Dataset: Vietnamese Alpaca (180 train samples, 20 eval samples)
• Training Framework: Unsloth + TRL SFTTrainer
• Quantization: 4-bit (QLoRA)

Metrics

• Trainable Parameters: 2,293,760
• Training Time: 4.01 minutes
• Peak VRAM: 7.00 GB
• Eval Loss: 1.6680
• Perplexity: 5.30

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "unsloth/Llama-3.2-3B-Instruct-bnb-4bit",
    load_in_4bit=True,
    device_map="auto"
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "luckyman2907/lab21-llama3.2-3b-r8")
tokenizer = AutoTokenizer.from_pretrained("luckyman2907/lab21-llama3.2-3b-r8")

# Generate
prompt = "Giải thích về machine learning"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

• Epochs: 3
• Learning Rate: 2e-4
• Scheduler: Cosine with warmup (10%)
• Batch Size: 8 (effective, via gradient accumulation)
• Optimizer: AdamW 8-bit
• GPU: Tesla T4 (16 GB)

Comparison with Other Ranks

Recommendation: Rank 16 offers the best balance between performance and efficiency.

Citation

@misc{lab21-lora-r8,
  author = {luckyman2907},
  title = {LoRA Adapter (Rank 8) - Llama 3.2 3B Vietnamese},
  year = {2026},
  publisher = {HuggingFace},
  howpublished = {\url{luckyman2907/lab21-llama3.2-3b-r8}}
}

License

Apache 2.0 (following base model license)