ft-llama-3.1-8B-capstone-personifAI-edu

Model Card for Model ID

This model has been fine-tuned to enhance its capabilities in generating personalized educational content, including audiobooks, textbooks, and quizzes on topics such as Data Analytics, Digital Marketing, and Project Management. It is designed to provide detailed, domain-specific outputs that can aid in learning and comprehension for students and professionals. This model has been created as a Capstone Project for NMIMS MPSTME University in Mumbai

Model Details

Model Description

This model is a fine-tuned version of the LLaMA-3.1-8B model, aimed at generating educational materials. Fine-tuned on a custom dataset focusing on data analytics, digital marketing, and project management, this model excels in generating personalized content for these specific domains. It can create detailed and contextually rich text, suitable for educational purposes such as textbooks, quizzes, and instructional content.

• Developed by: Aum Ghag
• Funded by [optional]: [More Information Needed]
• Shared by [optional]: [More Information Needed]
• Model type: Casual Large Language Model (LLM)
• Language(s) (NLP): English
• License: Apache 2.0
• Finetuned from model: meta-llama/Meta-Llama-3.1-8B

Model Sources [optional]

<!-- Provide the basic links for the model. -->

• Repository: [More Information Needed]
• Paper [optional]: [More Information Needed]
• Demo [optional]: [More Information Needed]

Uses

Direct Use

This model is intended for generating educational content, such as textbooks, quizzes, and audiobooks, across subjects like Data Analytics, Digital Marketing, and Project Management. It can be directly used by educators, content creators, and students to produce high-quality instructional material and study aids.

Downstream Use [optional]

The model can be fine-tuned further for specific educational purposes, such as developing customized curriculums, creating interactive learning modules, or generating content tailored to specific learning needs. It can also be integrated into educational platforms to enhance personalized learning experiences.

Out-of-Scope Use

This model is not suitable for generating content outside the educational domain, particularly in areas requiring specialized knowledge or where accuracy is critical, such as medical, legal, or financial advice. It should not be used for generating misleading information, spreading misinformation, or other malicious purposes.

Bias, Risks, and Limitations

This model was fine-tuned on a dataset specifically focused on educational content related to Data Analytics, Digital Marketing, and Project Management. As a result, it may exhibit biases present in the training data, such as:

• Educational Bias: The model's responses may reflect specific educational frameworks, methodologies, or regional practices prevalent in the source material. It may not fully align with other educational standards or curricula.
• Content Focus: The model is optimized for topics within Data Analytics, Digital Marketing, and Project Management. It may perform poorly or provide inaccurate information when queried on unrelated subjects.
• Cultural Bias: The dataset might include cultural or industry-specific perspectives that are not universally applicable, potentially leading to biased or one-sided outputs.

Recommendations

Users should take the following into account when using the model:

• Context Awareness: Ensure that the generated content is reviewed and adapted to fit the specific educational or professional context where it will be used.
• Human Oversight: Use human judgment to verify the accuracy and relevance of the content, especially when applied in diverse cultural or educational settings.
• Content Limitations: Be cautious when applying the model outside its intended domain. For other subject areas, complementary resources or models may be necessary to achieve accurate results.
• Bias Mitigation: Consider using additional datasets or fine-tuning steps to mitigate any identified biases if the model is to be used in broader or more sensitive contexts.

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("aumghag/ft-llama-3.1-8B-capstone-personifAI-edu")
model = AutoModelForCausalLM.from_pretrained("aumghag/ft-llama-3.1-8B-capstone-personifAI-edu")

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)

prompt = "What are Large Language Models?"
result = pipe(f"[INST] {prompt} [/INST]")
print(result[0]['generated_text'])

Training Details

Training Data

The model was fine-tuned on the aumghag/Data-Analytics-Digital-Marketing-Project-Management-QA_DB dataset, which includes instructional content in the fields of Data Analytics, Digital Marketing, and Project Management. This dataset consists of questions and answers designed to train the model for generating educational content.

Training Procedure

The model was fine-tuned using LoRA (Low-Rank Adaptation) and SFT (Supervised Fine-Tuning) techniques to specialize it in generating educational content. The training procedure involved a single epoch of fine-tuning with gradient checkpointing enabled to optimize memory usage. The model was trained on a 4-bit quantized version using the BitsAndBytes library to accelerate training on GPUs with limited memory.

Preprocessing [optional]

Preprocessing included tokenization using the LLaMA tokenizer, with padding set to the right side and the eos_token used as the pad_token. No additional preprocessing steps were applied to the dataset beyond tokenization.

Training Hyperparameters

• Training regime: Mixed precision using fp16 (16-bit floating-point precision) was utilized during training. The model was fine-tuned using the QLoRA method with a LoRA attention dimension of 64, LoRA alpha of 16, and a dropout of 0.1. Gradient accumulation was set to 1, and the learning rate was 2e-4 with a cosine scheduler. Weight decay was set to 0.001.

Evaluation

Testing Data, Factors & Metrics

Testing Data

The testing data used for evaluation included a split of 80/20 on the dataset aumghag/Data-Analytics-Digital-Marketing-Project-Management-QA_DB. The dataset comprises various question-answer pairs relevant to data analytics, digital marketing, and project management. For detailed information, refer to the associated Dataset Card.

Factors

The evaluation was disaggregated by the following factors:

• Task Domains: The model was evaluated on three key domains: Data Analytics, Digital Marketing, and Project Management. These domains were selected to assess the model's performance across diverse areas relevant to the target audience.
• Question Complexity: The evaluation considered the complexity of the questions, ranging from basic to advanced levels, to understand how well the model handles varying degrees of difficulty.
• Response Length: The evaluation also took into account the length of the generated responses to ensure the model provides adequate detail without being overly verbose.

Metrics

The following metrics were used to evaluate the model's performance:

• Accuracy: The correctness of the model's answers, particularly in generating factually accurate information.

• Perplexity: A measure of how well the model predicts the next token, which helps evaluate the fluency and coherence of the generated text.

• Relevance: Assessed by how well the generated responses align with the prompt and the domain-specific requirements.

• Diversity: Measures the variety of responses generated by the model, ensuring it doesn't repeat or generate redundant information.

• Human Evaluation: A manual review by subject matter experts to assess the overall quality and usefulness of the generated content.

Results

The model demonstrated strong performance in generating accurate and coherent text across the evaluated domains. The model delivered strong performance in generating Audiobooks, Personalized Textbooks, Quizzes (which target weak topics of students) and designing a curriculum for the students.

Summary

The fine-tuned LLaMA model demonstrates robust capabilities in generating high-quality, relevant, and accurate content across the domains of Data Analytics, Digital Marketing, and Project Management. The model excels in producing coherent and contextually appropriate responses, with a low perplexity score and high relevance. The model's strong performance across multiple metrics suggests that it is a valuable tool for users looking to automate content generation tasks.