falcon-7b-qlora-interview_qa-support-bot

Model Card for falcon-7b-qlora-interview_qa-support-bot

NOTE: Please review the repository for detailed instructions on how to use this model. Do not directly click the "Use this model" button; instead, access the model through the inference.py as mentioned in the repository.

This model has been fine-tuned for interview question answering and supports dynamic dialogue interactions. It uses PEFT (Parameter-Efficient Fine-Tuning) with bitsandbytes quantization for memory-efficient deployment.

Model Details

Model Description

This model is built on top of the Falcon 7B model from TII UAE and has been fine-tuned using PEFT for interview question answering support. The model is quantized using bitsandbytes to enable 4-bit precision, making it suitable for deployment on lower-resource environments while maintaining performance.

• Developed by: Katakam Pranav Shankar
• Model type: Causal Language Model (for Q&A and dialogue systems)
• Language(s): English
• Fine-tuned from model: tiiuae/falcon-7b'
• Research Paper: https://www.mlexpert.io/blog/fine-tuning-llm-on-custom-dataset-with-qlora#training ]

Model Sources

• Repository: https://huggingface.co/Pranav06/falcon-7b-qlora-interview_qa-support-bot

Uses

Direct Use

This model is designed to generate interview-related responses, allowing users to engage in an interactive interview process. It supports multiple rounds of questions and responses and can handle complex dialogue.

Downstream Use

This model can be integrated into larger systems for dynamic Q&A support, such as customer service bots, recruitment tools, or interview preparation platforms.

Out-of-Scope Use

• The model is not intended for generating harmful, biased, or inappropriate content.
• It should not be used in real-world interviews without human supervision, as it may still generate responses with inaccuracies.

Bias, Risks, and Limitations

As with all language models, there may be biases in the model due to the data it was trained on. We encourage users to review the output carefully, especially when it comes to sensitive interview-related topics.

Recommendations

Users should be aware of the model's limitations, especially in handling nuanced or highly specialized questions. It is recommended to review and supervise the model's output before deploying in production environments.

How to Get Started with the Model

To use this model with PEFT and bitsandbytes quantization, follow the steps below:

1. Install Required Libraries:

   Before you start, make sure you have the necessary dependencies installed:

   pip install transformers bitsandbytes torch peft
   

2. Download the Custom Script:

   You can use the custom inference.py script for efficient deployment with PEFT and quantization. This script ensures that the model is loaded properly and ready for generating responses.

   Download the inference.py script from the model repository, then run the script as follows:

   python inference.py
   

3. Example Code to Use the Model:

   Here's an example of how to use the model for generating a response:

    import json
    import torch
    from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
    from peft import PeftConfig, PeftModel
    import bitsandbytes as bnb
    import os
    
    # Set up CUDA for GPU use
    os.environ["CUDA_VISIBLE_DEVICES"] = "0"
    
    # Configuring bitsandbytes quantization
    bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_use_double_quant=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16,
    )
    
    # Load the PEFT model configuration and model
    PEFT_MODEL = "Pranav06/falcon-7b-qlora-interview_qa-support-bot"
    config = PeftConfig.from_pretrained(PEFT_MODEL)
    
    model = AutoModelForCausalLM.from_pretrained(
        config.base_model_name_or_path,
        return_dict=True,
        quantization_config=bnb_config,
        device_map="auto",
        trust_remote_code=True,
    )
    
    # Load the tokenizer
    tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
    tokenizer.pad_token = tokenizer.eos_token
    
    # Apply PEFT to the model
    model = PeftModel.from_pretrained(model, PEFT_MODEL)
    
    # Genration configuration
    generation_config = model.generation_config
    generation_config.max_new_tokens = 200
    generation_config.temperature = 0.7
    generation_config.top_p = 0.7
    generation_config.num_return_sequences = 1
    generation_config.pad_token_id = tokenizer.eos_token_id
    generation_config.eos_token_id = tokenizer.eos_token_id
    
    DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
    
    # Define a function to generate responses
    def generate_response(question: str) -> str:
        prompt = f"""
    <human>: {question}
    <assistant>:
    """.strip()
        encoding = tokenizer(prompt, return_tensors="pt").to(DEVICE)
        with torch.inference_mode():
            outputs = model.generate(
                input_ids=encoding.input_ids,
                attention_mask=encoding.attention_mask,
                generation_config=generation_config,
            )
        response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    
        assistant_start = "<assistant>:"
        response_start = response.find(assistant_start)
        response = response[response_start + len(assistant_start) :].strip()
        response_lines = response.split("\n")
        final_response = response_lines[0].strip()
    
        return final_response
   
   
     # Example usage
     input_text = "Tell me about yourself."
     response = generate_response(input_text)
     print(response)
   

Training Details

Training Data

The model was fine-tuned on approximately 500+ interview-related question-answer pairs and dialogue examples.

Training Procedure

The model was fine-tuned with PEFT using a combination of supervised learning from Q&A datasets.

Training Hyperparameters

• Training regime: Mixed precision (fp16)
• Optimizer: AdamW
• Batch Size: 32

Evaluation

Testing Data, Factors & Metrics

• Metrics: The model's accuracy is evaluated based on how well it answers interview questions across different topics. Metrics such as F1-score and accuracy are used to assess the model's performance.

Results

The model performs well in generating coherent and contextually accurate responses for interview-style questions. However, its performance can degrade when faced with highly technical or niche questions.

Technical Specifications

Model Architecture and Objective

The model is based on the Falcon-7B architecture with PEFT fine-tuning. It is designed to handle natural language understanding and generation tasks, especially for interview Q&A systems.

Compute Infrastructure

The model was trained using GPUs and optimized using 4-bit quantization with the bitsandbytes library to reduce memory usage.

Citation

If you use this model in your research or application, please cite the original Falcon 7B model as follows:

BibTeX:

@misc{falcon7b,
  author = {TII UAE},
  title = {Falcon 7B},
  year = {2023},
  publisher = {Hugging Face},
  url = {https://huggingface.co/tiiuae/falcon-7b}
}

More Information

For additional details, including fine-tuning or usage instructions, please visit the model repository.

Model Card Authors

• Katakam Pranav Shankar

Model Card Contact

For any questions regarding the model, please contact [email protected].