TinyLlama-SFT-Alpaca

TinyLlama-SFT-Alpaca

TinyLlama-1.1B-Chat fine-tuned using Supervised Fine-Tuning (SFT) with LoRA on the yahma/alpaca-cleaned dataset as part of IBA Karachi NLP with Deep Learning Assignment 4.

Model Details

• Developed by: Hadeeqa Al Islam
• Model type: Causal Language Model
• Language: English
• Finetuned from: TinyLlama/TinyLlama-1.1B-Chat-v1.0
• Training data: yahma/alpaca-cleaned (20,000 samples)
• LoRA config: r=8, lora_alpha=16, target_modules=[q_proj, v_proj]
• Training: lr=2e-4, batch=4, epochs=2, fp16=True

How to Get Started

tokenizer = AutoTokenizer.from_pretrained("Haldi247/TinyLlama-SFT-Alpaca") model = AutoModelForCausalLM.from_pretrained("Haldi247/TinyLlama-SFT-Alpaca") messages = [{"role": "user", "content": "What is photosynthesis?"}]

Training Details

• Dataset: yahma/alpaca-cleaned (20,000 samples after filtering)
• Hardware: NVIDIA RTX 5070 Ti (16GB VRAM) via WSL2
• Training time: ~14 minutes

Training Procedure

Training Hyperparameters

• Training regime: fp16 mixed precision
• Learning Rate: 2e-4
• Batch Size: 4
• Epochs: 2

Evaluation

Testing Data, Factors & Metrics

Metrics

• BLEU Score: Used to evaluate the overlap of n-grams between the model output and the ground truth.
• BERTScore: Used to compute semantic similarity between the generated text and reference text using BERT embeddings.

Results

• Average BLEU Score: 0.4303
• Average BERTScore: 0.7236

Limitations

• Tokenization Constraints: The training process used sequence packing to manage memory effectively. Because the model was trained with packed sequences without special tokens (to prevent cross-contamination), it may exhibit formatting issues if the inference environment does not strictly adhere to the expected chat template.
• Data Bias: The model is fine-tuned on the Alpaca dataset, which may inherit biases present in the synthetic instruction data.
• Inference Stability: Due to the aggressive packing strategy, users may observe repetition or formatting artifacts if the prompt structure deviates significantly from the training format.

Summary

Model Examination [optional]

<!-- Relevant interpretability work for the model goes here -->

[More Information Needed]

Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: [More Information Needed]
• Hours used: [More Information Needed]
• Cloud Provider: [More Information Needed]
• Compute Region: [More Information Needed]
• Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

[More Information Needed]

Compute Infrastructure

[More Information Needed]

Hardware

[More Information Needed]

Software

[More Information Needed]

Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

BibTeX:

[More Information Needed]

APA:

[More Information Needed]

Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->

[More Information Needed]

More Information [optional]

[More Information Needed]

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

[More Information Needed]