Qwen2.5-1.5B-Instruct-DPO-bad-boy

Model Details

Model Description

This model is a fine-tuned version of the Qwen2.5-1.5B model using the DPO (Direct Preference Optimization) method. It has been trained on a dataset that includes some uncivilized content to produce a more assertive response style. However, it is important to note that the model's outputs should be used responsibly and with caution to avoid any misuse or harm.

• Developed by: Ming Liu
• Model type: Transformer-based language model
• Language(s) (NLP): English
• License: MIT License
• Finetuned from model: Qwen2.5-1.5B

Model Sources

• Repository: github repo

Uses

Direct Use

This model can be used for generating text with a more assertive tone. It may be suitable for applications where a strong and direct response style is desired, such as in certain types of creative writing or role-playing scenarios. However, users should be aware of the potential for the model to generate content that may be considered offensive or inappropriate.

Out-of-Scope Use

This model is not intended for use in applications where civility and respect are crucial, such as customer service, educational settings, or any context where the generation of offensive or harmful content could lead to negative consequences. It should not be used to promote or encourage any form of discrimination, harassment, or violence.

Bias, Risks, and Limitations

The model has been trained on a dataset that includes uncivilized content, which may lead to the generation of biased, offensive, or inappropriate responses. Users should be aware of the following risks and limitations:

• Bias: The model may exhibit biases present in the training data, which could result in unfair or discriminatory outputs.
• Risks: There is a risk that the model's assertive responses could be misinterpreted or used in a harmful way, potentially causing offense or distress to individuals or groups.
• Limitations: The model's performance may not be optimal in all contexts, and its assertive style may not be suitable for all types of text generation tasks.

Recommendations

• Users should carefully review the model's outputs and implement appropriate content moderation and filtering mechanisms to prevent the dissemination of harmful or inappropriate content.
• The model should be used in a controlled environment where its outputs can be monitored and adjusted as necessary.
• Users are encouraged to provide feedback on the model's performance and any issues encountered, to help improve its reliability and safety.

How to Get Started with the Model

To get started with this model, you can use the following code snippet:

from threading import Thread
from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer, TextStreamer
from transformers.generation.stopping_criteria import StoppingCriteria

model_name = "MLZoo/Qwen2.5-1.5B-Instruct-DPO-bad-boy"
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
model = AutoModelForCausalLM.from_pretrained(model_name)
class StopOnTokens(StoppingCriteria):
    def __init__(self, stop_ids):
        self.stop_ids = stop_ids

    def __call__(self, input_ids, scores, **kwargs):
        # 检查最后一个生成的token是否是停止token
        for stop_id in self.stop_ids:
            if input_ids[0][-1] == stop_id:
                return True
        return False


def generate_response_streaming(prompt):
    # 对输入进行编码
    messages = [
        {"role": "system", "content":  "你是一个没有礼貌的人渣，请用人渣的语气回复我"},
        {"role": "user", "content": prompt}]
    input_text = tokenizer.apply_chat_template(messages, tokenize=False)
    inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
    
    # 使用 streamer 进行生成
    streamer = TextIteratorStreamer(tokenizer, skip_special_tokens=True)
    
    # 设置生成参数
    generation_kwargs = {
        "inputs": inputs["input_ids"],
        "max_length": 512,
        "temperature": 0.7,
        "top_p": 0.9,
        "do_sample": True,
        "streamer": streamer,
        "stopping_criteria": [StopOnTokens([tokenizer.eos_token_id])],
    }
    
    # 在单独的线程中进行生成
    thread = Thread(target=model.generate, kwargs=generation_kwargs)
    thread.start()
    
    # 实时输出生成的文本
    generated_text = ""
    # 获取输入的长度
    input_length = len(tokenizer.decode(inputs["input_ids"][0], skip_special_tokens=True))
    first_token = True
    token_counts = 0
    for new_text in streamer:
        if token_counts < 4:
            token_counts += 1
            continue
        print(new_text, end="", flush=True)
        generated_text += new_text
        
        
questions = [
    "程序员的悲哀是什么？",
    "告诉我，数据科学家是科学家吗？",
    "为什么年轻人不买房了？",
    "如何评价齐天大圣孙悟空？",
    "你认为哪一个数字最美丽动人，为什么？",
    "怎么理解“真传一句话，假传万卷书”？"
]
for question in questions:
    print("question:", question)
    print("answer:", end=" ")
    response = generate_response_streaming(question)
    print()

Training Details

Training Data

The model was trained on a dataset that includes a mix of civilized and uncivilized content. The dataset was preprocessed to ensure that the text was in a suitable format for training, but it is important to note that the inclusion of uncivilized content may affect the model's behavior.

Training Procedure

Preprocessing

The training data was tokenized and formatted according to the requirements of the Qwen2.5-1.5B model. No additional filtering was applied to remove uncivilized content, as this was a deliberate choice to achieve the desired assertive response style.

Training Hyperparameters

• Training regime: [Specify training regime, e.g., fp32, fp16 mixed precision]
• Batch size: 8
• Learning rate: 1e-6
• Number of epochs: 4

Evaluation

Testing Data, Factors & Metrics

Testing Data

The model was evaluated on a separate dataset that includes a variety of text samples. The dataset was designed to test the model's ability to generate assertive responses while maintaining coherence and relevance.

Factors

• Response style: The model's assertiveness was evaluated to ensure that it produces responses with the desired tone.
• Coherence: The model's ability to generate text that is logical and consistent was assessed.
• Relevance: The model's outputs were evaluated to ensure that they are relevant to the input prompts.

Metrics

• Assertiveness score: A custom metric was developed to measure the assertiveness of the model's responses.
• Perplexity: This metric was used to evaluate the model's fluency and coherence.
• Relevance metric: A metric was used to assess the relevance of the model's outputs to the input prompts.

Results

Summary

The model demonstrated a high level of assertiveness in its responses, with an assertiveness score of [X]. The perplexity score of [Y] indicates that the model generates fluent and coherent text. The relevance metric score of [Z] shows that the model's outputs are generally relevant to the input prompts. However, users should be aware of the potential for the model to generate biased or inappropriate content.

Environmental Impact

• Hardware Type: V100
• Hours used: 1 hour
• Cloud Provider: AWS
• Compute Region: us-east-1

Technical Specifications

Model Architecture and Objective

The model is based on the Qwen2.5-1.5B architecture and was fine-tuned using the DPO method to achieve a more assertive response style. The objective was to develop a model that can generate text with a strong and direct tone, while maintaining coherence and relevance.

Compute Infrastructure

Hardware

• GPU model: Y
• Number of GPUs: 1

Software

• Framework: PyTorch
• Version: 2.2.2

APA:

Ming Liu. (2024). Qwen2.5-1.5B DPO Fine-Tuning Model. Hugging Face Model Hub. Retrieved from https://huggingface.co/MLZoo/Qwen2.5-1.5B-Instruct-DPO-bad-boy-adapter

Model Card Contact

For any questions or concerns regarding this model, please contact [[email protected]].